12

Possible Duplicate:
My server's been hacked EMERGENCY

Last weekend my company's site was hacked.

They did the nicest thing of doing that on a Friday evening so we only noticed the attack on Monday morning.. The funny thing is that we switched from Windows to Linux recently because it was supposed to be more stable and secure. Go figure. And yes, we got us blacklisted on Firefox and Chrome.

Since I am not a Linux expert, I am looking for advice on how to avoid problems like this in the future. What steps do you take to protect your systems? It seems we had weak passwords, but shouldn't Linux block the account after a few failed logins? They tried more than 20 combinations...

In addition to that, I am looking for a tool (or service) similar to pingdom but applied to security. If my site is ever hacked, alert me. Is that such a thing? A Hacking monitor? :)

Another thing, how do you notify your clients about such issues? Do you just ignore and hope no one noticed? Email explaining what happened?

*posting as anonymous to avoid more bad exposure to my company, which is bad already...

Community
  • 1

11 Answers11

23

An operating system is not "stable and secure". A properly configured and well-administered infrastructure can be made more secure than one that isn't, but security isn't a boolean. You can't "buy security" or "install security" by using a particular product / technology.

You're not a "Linux expert", so it makes sense that you need to hire / contract with someone who can configure your servers well, from a security perspective. It's not something you can do one and "be done". Patches are being released all the time for newly found vulnerabilities and bugs. If you don't have an employee who has the job of keeping up you really need to consider subscribing to some type of "managed" service to maintain your server computers. This is an ongoing concern, and needs to be factored into the budget / TCO of the system as a whole.

There are "hacking monitors", to some extent. Intrusion detection systems (IDS), intrusion prevention systems (IPS), etc, fill that niche. It's an arms race, though. You can't just purchase an off-the-shelf IDS/IPS product, pay somebody to put it in, then sit back and feel smugly secure. Just like keeping operating system software and application software patched, the "hacking monitor" infrastructure must be kept up to date, too.

You need to talk to a lawyer. You may have clients located in places where you are bound to disclose such occurrances by law. Even if you're not, it sure seems slimy to me not to let your clients know if their data was placed at risk. There's the damage to "your good name" now in disclosing the hack, but there's a multiplicity of that damage if it comes out later that you tried to cover it up-- especially if you're breaking the law by doing it.

Practical Stuff:

Your hacked machine(s) are trash. They need to be reloaded from a known-good backup or, better yet, reloaded from clean OS binaries and re-populated with data. This is like "malware cleanup" except worse, because your adversary is much more likely a thinking being instead of a dumb piece of software (though you may have been hacked by a 'bot). The chance that there are "back doors" in your servers is real.

The data on the server computers should be considered to have been disclosed to the public. Even if it's not now, it could be.

Any credentials to other computers stored on the hacked computers are public. Start getting those passwords to other computers changed NOW and make sure that the other computers are intact. (Does anybody user Tripwire anymore? That'd sure be nice in this occasion...)

You've got a mess. Handle it well and you will come out better. Handle it poorly and, next time, you may not have a company.

In the future, you should be using strong authentication and encrypted management protocols (SSH, public-key based authentication). I've already suggested that you get a "Linux expert", even if it's just on contract, to get you started down the right track. I can't encourage that enough. You'll get to see, with this breach, how that would have "paid for itself".

All the common stuff applies:

  • Don't run services you don't need to.
  • Disable default credentials.
  • Follow the principle of least privilege.
  • Have tested, offline, and off-site backups.
  • Know your legal requirements re: breach disclosure.
  • Keep your systems / applications updated.
KPWINC
  • 11,274
  • 3
  • 36
  • 44
Evan Anderson
  • 141,071
  • 19
  • 191
  • 328
19

It seems we had weak passwords...

...we switched from Windows to Linux recently because it was supposed to be more stable and secure. Go figure.

Weak passwords are platform independent.

Linux is more flexible than Windows in many scenarios, and thus can be made more secure when those certain situations arise. Switching from Windows to Linux for no real reason, especially when you're unfamiliar with the environment, is a bad call. If you run an internet-facing server and don't understand how the services on that server work, whether it be Windows, Solaris, RHEL, or BSD, you're asking for trouble.

As for how to tell your clients, if any of their data was exposed or even POTENTIALLY exposed, AT ALL, call them ASAP. No email, no hope it goes away. Use your phone that's sitting on your desk.

Besides legal repercussions, you were providing a service to them that they were undoubtedly using to provide services for others in some way. You owe it to them to disclose any potential breach of data so they can adjust their workflow accordingly and notify anyone else downstream from them that it may affect.

MDMarra
  • 100,183
  • 32
  • 195
  • 326
  • Randomly a downvote and no explaination after almost a year? I'd love for the downvoter to comment so that I can see what he/she disagrees with here. – MDMarra Jun 03 '10 at 15:29
14

Linux like any platform is only as secure as the people who administrate the systems. If you were more experienced with Windows then moving to Linux probably wasn't the best idea. You can setup Windows to be just as secure as Linux provided that you take the correct steps to secure the environment which includes firewalls between the public internet and your internal network, and secure VPNs for connecting from your home to the internal network.

Depending on the level of the breach will determine what you tell your customers. If no customer data was taken then you can give your customers just some basic information about what happened. However if Customer data was accessed you now have some state notification laws to deal with depending on the state that your company is in, and the states your customers are in.

mrdenny
  • 27,074
  • 4
  • 40
  • 68
11

The funny thing is that we switched from Windows to Linux recently because it was supposed to be more stable and secure. Go figure.

Go figure, indeed. You switched to a system you have little experience in, and you got bitten in the ass. The same thing would happen if you bought a high powered sports car after riding a bike all your life - it may be more powerful, but in the wrong hands it's also dangerous.

It seems we had weak passwords, but shouldn't Linux block the account after a few failed logins? They tried more than 20 combinations...

It can easily be configured to do so, by add-ons such as denyhosts. Knowing to install such things is one of those bits where being an expert in the systems you're using is useful.

In addition to that, I am looking for a tool (or service) similar to pingdom but applied to security. If my site is ever hacked, alert me. Is that such a thing? A Hacking monitor? :)

There are some intrusion detection apps out there, but new methods of cracking a server appear all the time. Nothing will be 100% reliable in detecting successful intrusions.

Another thing, how do you notify your clients about such issues? Do you just ignore and hope no one noticed? Email explaining what happened?

Honesty will serve you better in the long run. Companies that are honest, proactive, and communicative weather nastier storms than companies that hide things from their customers. Outages can be survived, but not if the people paying you feel scammed.

ceejayoz
  • 32,469
  • 7
  • 81
  • 105
6

The hacking of your site likely had very little to do with the underlying operating system, and more to do with the code running on your site. All it takes is a single SQL injection, and you're history.

Since you were blacklisted by Google, I'll assume someone managed to set up a malicious script on your server, in which case you might try something like mod_security, it's not a piece of cake to configure, but it's worth a try. It is, however, of paramount importance to ensure your code is free of these kinds of vulnerabilities.

On the other hand, if it was a vulnerability in your operating system, you may want to try switching distributions to something intended for web serving, like FreeBSD, CentOS or RHEL, assuming you aren't already using one. You may want to consider beefing up SELinux, or adding an Intrusion Detection/Prevention System of some kind aswell.

In many places, it is a legal requirement to notify your customers of security breaches if personal information was potentially compromised, you may want to look into that.

Some more details would be helpful in answering your question more to your specific situation.

Ehtyar.

Ehtyar
  • 802
  • 6
  • 13
6

Others have already covered most of the important points. But to add to what they have said security is only as strong as the weakest link in the chain. This is almost always the human element involved.

Assuming that this was a brute force ssh attack if it only took them 20 tries then your passwords were pretty much non-existent. These attacks are almost always automated attacks and are pretty easy to block with the following iptable rules.

sudo iptables -A INPUT -i eth0 -p tcp --dport 22 -m state --state NEW -m recent --set --name SSH  
sudo iptables -A INPUT -i eth0 -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 3 --rttl --name SSH -j DROP

These rules will allow 3 ssh connections from an ip address on eth0 in 60 seconds and then block that ip address for 60 seconds. This is normally enough to cause the automated attack to move on to another host.

You will need to make sure that these iptable rules are restored when you reboot. Different distributions have different ways of doing that so google that or let us know what distro you're running and someone can provide you with that information.

3dinfluence
  • 12,409
  • 2
  • 27
  • 41
  • 2
    Fail2ban (or denyhosts as also suggested) will deal with brute force attacks in a similar fashion. Blocking potential attackers is the best way to secure your network imo. – theotherreceive Jul 03 '09 at 01:27
6

As a Windows admin who uses Linux for certain jobs I'd just like to add that you made a very fundamental mistake. You switched OS because of Linux's reputation for stability and reliability and expected it to take over your responsibility. On my test machine it takes nearly 12 hours to crack my current Windows password. I haven't tested my Linux password but it's equally complex (but not the same). The point is that I don't take lesser measures simply because the OS is more secure. If anything, I'm even more careful with Linux because I have less experience with it and am therefore more likely to overlook something.

John Gardeniers
  • 27,262
  • 12
  • 53
  • 108
2

As far as a service similar to pingdom, but applied to security, I will suggest Sucuri's free Network integrity monitor.

What it does? It monitors your web site (and domains) on real time and alert you if they are ever defaced, blacklisted, hacked, etc. Link: http://sucuri.net

As the name implies, it monitors the integrity of your 'internet' presence.

*disclaimer: I developed it.

sucuri
  • 2,817
  • 1
  • 22
  • 22
  • 4
    Only stupid hackers will deface a front-page. Better to hide the activity somewhere obscure and avoid detection, unless the goal is direct interference with the target. – jldugger Jul 03 '09 at 04:00
  • 3
    Only stupid hackers AND people defacing the site in order to make a statement (front page = statement seen by more people) AND people injecting malware into the site (font page = more possible drive-by installs). Testing the front page won't catch everything, but it will catch some problems. Of course if the front page is dynanic then scanning for changes will do nothing as there will almost always be a change. – David Spillett Jul 03 '09 at 07:42
  • 1
    Perfect reply by David. jldugger is missing the point that most attacks nowadays are automated to inject malware, hijack a domain or just deface for fun/political reasons. – sucuri Jul 03 '09 at 15:17
  • 2
    +1 thanks for contributing your time to the development of something useful and free of charge. – cop1152 Jul 12 '09 at 19:56
  • Only stupid hackers waste their time choosing a target site or page. Most of these hacks are automated crawlers, looking for ways to inject stuff into SQL. Once they find something, it's further automated to list all the tables (a good reason to disallow the web client db account to query this meta data) and to search through the table's text entries to match that to text on the page. Then it inserts something like 'OriginalText"> – dlamblin Jul 12 '09 at 21:35
1

I always liked the advice: close all ports, dont let any traffic through, either way, at all. Then, on an as-needed basis, gradually open only the ports that are needed and nothing more.

While this wont necessarily keep you safe, it will keep you familiar with your system and ensure that only the bare minimum is open.

cop1152
  • 2,626
  • 3
  • 21
  • 32
0

Since you didn't specify what exact type of a "hack" it was, I think I can assume you got compromised through SSH brute force. Here are some general tips:

  • Use SELinux or AppArmor to restrict access per process, especially apache and mysql
  • Use mod_evasive and mod_security on apache
  • Don't run SSH on default port
  • Disable root login through ssh, and use a very strong root password (such as: 6uF3ceDa7u, bRusteth6F, breGUfE4aT, 7Ub8R9Then)
  • Don't allow password login (use keys) -OR- use ssh-guard, blacklists and iptable rules to stop repetitive access
LiraNuna
  • 291
  • 2
  • 16
  • 4
    Moving ssh from port 22 is stupid. It won't stop anyone who really wants to get access to your system, instead it just provides a false sense of security via obscurity. – theotherreceive Jul 03 '09 at 01:24
  • 2
    No, it's not stupid. It increases the amount of time an attacker needs to make that initial SSH connection by a factor on the order of 100-1000. Now, if someone is specifically focused on *your* server, then sure it doesn't make a difference, but hackers often scan large numbers of servers and they won't have the resources to search all ports on all of them. There's no harm in keeping yourself from being one of the easy targets. (disclaimer: switching the SSH port is of course no excuse to skimp on other security practices.) – David Z Jul 03 '09 at 02:11
  • Fail2ban, denyhosts, and other similar methods will equally cause scanning scripts to quickly get bored and move off, without the annoyance or obscurity factors of moving ports. – theotherreceive Jul 03 '09 at 02:26
  • 1
    When using a non-standard port any attempt to locate it is a port scan and therefore *should* trigger a defensive system. – John Gardeniers Jul 03 '09 at 04:21
  • Those were just general tips - the attack was most likely automated, and bots will just move on if connection is refused from port 22. I didn't say it's a life saver. – LiraNuna Jul 03 '09 at 04:30
  • +1 to David's "it's not stupid". Yes of course it IS SBO, but so many attacks don't care for which target and just look for the low-hanging fruit. – Cheekysoft Jul 03 '09 at 13:41
  • The bots may ignore SSH on port 50000, but it introduces more complications for the already clueless people admining the server and you can reap the same benefits with a variety of tools that work quite well. – duffbeer703 Jul 13 '09 at 03:52
0

I've posted some comments on the various bits of specific advice that has been offered to try to avoid future attacks. I found myself in a similar situation just over a year ago, and certainly analysing the attack to improve your setup is important, but imo right now you should be worried about the servers you have running. How do you know you can trust anything on them? If you can't, then you need to wipe them.

theotherreceive
  • 8,235
  • 1
  • 30
  • 44