How I can prove that SSH attacks are not from my application?

Question

My ruby on rails web application is hosted on Linode. Linode opened a ticket and blamed my Linode server for attacking other servers. The logs revealed that it's a SSH brute force attack which originated from my server. I have studied my source code, checked if there is any malicious script on my server but I found none.

1. How can I prove that attack is not originating from my end?
2. If however the attack is originating from my end, how can I detect which process is responsible for the attack?

My web application is running on Ubuntu 12.04.

Answer 1

Linode uses Xen, which means that you not only have an app; you have a whole operating system, complete with kernel and base libraries. The attack comes from your system, so it is completely in your jurisdiction. The attacker entered your system in some way, but once there, if he is at least half competent, he used a rootkit to transform his break-in into a permanent installation. The downside of it is that a rootkitted system will not show up from the inside of the system: the kernel would have been infected and will not show the attack process from the attackers.

(I am using here the assumption that once the attacker entered, he could escalate his rights into root level -- that's the prudent assumption, because operating systems are rarely rock solid against local attackers.)

This implies that your system should be wiped out and reinstalled from scratch; it has been compromised and can no longer be trusted. At that point, you would have two goals:

1. Understand how the attacker came in, so that he won't do it again on the newly reinstalled system.
2. Convince the Linode people that you are a victim, not an accomplice.

For the first point, this is a matter of analysis of all the logs and other traces that can be found in your system. So before scraping it, take a full copy of it (preferably, a byte-by-byte copy of its virtual hard disk) for ulterior analysis. But remember that if the attacker is reasonably good, he covered his tracks (that's what rootkits are for). Also, some auditing of your application code would be in order: the attacker entered in some way through one of the externally accessible services of your server, which includes your application. Another possibility is that the attacker guessed your SSH account password (after all, you know that this attacker does run brute-force attacks on SSH logins): on your new system, use a new, bigger, more random password (don't type it on your current system ! It is compromised, thus possibly grabbing password changes on-the-fly).

For the second point, this is a matter of trust and human relationships. You cannot prove that you are not the attacker; but the Linode people know that hijacked systems are an unfortunately common reality. Just exercise due diligence, such as reinstalling the system from scratch: going through that tedious task will go a long way towards establishing your good faith in that matter.

Answer 2

You need to study the traffic coming from your server. It will reveal if your server is connecting to SSH services on other networks.

If you find no such evidence from the traffic generated from your server offer this proof to linode and have them either refute it or accept it.

If you find evidence you need to go through network traffic, logs, running processes and local files to determine what causes your server to behave this way.

When this is done you need to nuke the server and reinstall from a trusted source while also implementing the fix before it is exposed to the same attack vector that compromised it in the first place.

Answer 3

You can prove it, if you can find a compromised system of any way, shape or form on your server. You have a part of the proof. Having a single line of a log file is not proof. So, this is really hard to "proof". IT Security has a lot to do with the WOT (Web of Trust) You need to let them "trust" you on some way. The easiest way do to this is to hand over the source code. But this is not possible in most cases.

But still, if they have hard proof that your server has tried multiple logins on there servers with SSH you might have something going on. I'd make an image (harddrive, RAM) reboot, and hire a pentester.

About detecting the attack yourself. Put a Sniffer on the ethernet port of your server. The best way to do this is to use a SPAN-port. Do not sniff on the system itself.

So goodluck :)