Debian security: lots of connections on port 80

I have a debian server (kernel: 2.6.32-5-amd64).

I normally run a jetty server on it, but lately it started getting tons of connections to it. It shouldn't get all this traffic, since its a pretty unknown server.

Running:

netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

Outputs hundres of IP's. I tried adding them all to iptables drop list, but new IP's keeps showing up.

I then went ahead and stopped Jetty, and all connections where gone. To make sure this was not a bug/security hole in Jetty, I fired up apache2, and all the connections started right away.

I looks like people are using it as a proxy server, using urlsnarf its showing tons of outgoing requests to Forums, ad sites, and you name it. Its doing so many request, that the CPU is jumping up and down, and eventually the server ends up crashing.

Does anyone know how they can do this? It seams like whatever server is listing on port 80, this is immediately begins.

Is this a DDOS attack? How are people using my server as proxy, only with software listing on port 80?

I have hostsdeny installed and deflate (http://deflate.medialayer.com/), but still the problem persists.

If you suspect or have any idea how to secure and fix this problem, I would be very thankful.

If I need to provide more data, let me know.

Thanks in advanced

Bolli

Posted 2013-06-08T22:04:23.463

Reputation: 382

Answers

This is not a DDOS attack if there is real traffic going on through your server.

What you are describing shouldn't be possible, but hackers may still have found a way. If your server was compromised, then it is much more likely that the attack came from inside your network via another infected computer.

I would suggest to reformat this server's disk and reinstall all software. Ensure that it is firewalled from both the external and internal networks.

You should also verify all the computers in your internal network that have access of any kind to this server, and in the future restrict more any such access.

Follow the the articles below for Apache (more info is surely to be found elsewhere):

Security Tips - Apache HTTP Server
20 ways to Secure your Apache Configuration

There are many articles for hardening Linux, so here are just a couple :

20 Linux Server Hardening Security Tips
Red Hat Linux Server Hardening Checklist

harrymc

Posted 2013-06-08T22:04:23.463

Reputation: 306 093

Thanks harrymc - actually I did try reinstalling the server, with the latest Ubuntu (its a Virtual machine, so I created a new server and allocated the static IP to it). I then started Jetty server on port 80 again (only with its default server html page) - and the requests starting flooding again. I also tried removing Jetty and installing apache instead, but as soon as I start apache2 service (again only with the default "it works!" page), the requests starts hammering ind again. – Bolli – 2013-06-23T14:29:12.573

Is this VM directly connected to the Internet with a fixed IP? Are the requests from the Internet or internal? – harrymc – 2013-06-23T14:30:59.037

It's directly connected with a static/fixed IP. The requests are from all around the world, I guess from other compromised servers. As said in my other comment, Wireshark shows that all these requests are SYN requests, so Im wondering if I am getting SYN Flooded. I have applied these steps to prevent it: http://pierre.linux.edu/2010/04/how-to-secure-your-webserver-against-syn-flooding-and-dos-attack/

But still I dont understand how they can do GET requests to external servers, with my server only listing on PORT 80?

– Bolli – 2013-06-23T14:34:40.477

And adding these steps to avoid SYN Flooding, has not helped. Only thing that works is to manually block the IP's in iptables, but new IP's keeps showing up. I was planing on running this as a production server in near future, so I can't simply auto-block them all. Unless there is a way to separate these requests from good-minded/normal requests? – Bolli – 2013-06-23T14:39:28.720

Funny that anybody would attack your server, unless you are just collateral damage. You could in iptables block-all except specific IPs to specific ports. See also this article.

– harrymc – 2013-06-23T15:18:56.190

Thanks - like I said I would like to use the server/ip for production in near future, so blocking all except specific IP's would not work for me. It is very funny that anyone would wan't to attack this server, since only 3 people knows about it. I suspect that the IP i brought has been used before or somehow ended up on a botnet list or something. I don't know, I'm just very curios how to fix this and how they are doing it. I have applied some of the iptables rules from your link - still not blocking all the requests :/ – Bolli – 2013-06-23T15:34:12.633

Syn attacks are very hard to avoid if done well, and possibly even your IP segment is hit with SYN scanning. You could try getting another IP or a cloud front-end in another IP segment. Or try to find a common factor between the SYN requests (such as port number) that will help set up a blocking rule. Some advanced commercial firewalls can protect the server against SYN attacks but are quite costly, and so are professional routers.

– harrymc – 2013-06-23T16:23:13.160

1The possibilities I see are that the previous owner of this IP (1) was running a proxy service, (2) was heavily into P2P, (3) or part of a network such as TOR, (4) or was infected by a botnet and served as a messenger node, (5) or that this IP segment is over-scanned by a botnet. It is quite possible that this previous owner also asked to have his IP changed, or had it revoked for illegal activities, and that's how you got it ... – harrymc – 2013-08-06T06:09:14.260

It's not really on topic but I would advise updating your kernel as 2.6.32-5 is vulnerable to a local root explot.

But your server could of been compromised already and being used as a proxy server for someone, if you're hosting a website have a look through it see if they are any suspicious looking pages.

Also install anti-rootkit software just incase.

Typically DDoS attacks would just show up as SYN requests if you looked at traffic through a program such as wireshark

user2341069

Posted 2013-06-08T22:04:23.463

Reputation: 38

Thanks for your answer. Wireshark shows a lot of SYN and ACK requests. It also shows a lot of GET requests to all kind of strange/spammy URL's. What I don't understand is how they can do outgoing GET requests, when <insert any software> listins on port 80. If I shut down the software listining on port 80, everything stops. – Bolli – 2013-06-23T14:18:20.237

The ACKs mean that this is most likely not a DDoS attack, with a full re-install these requests are still showing? – user2341069 – 2013-06-23T15:23:46.933

Yes - its a virtual server (see my comments on harrymc's answer), so I did try create a new VPS (with the latest Ubuntu instead of Debian) and assign the same static IP to it, while the original server was shutdown. Same thing happens again. Wireshark seams to show more SYN requests then ACK requests though. – Bolli – 2013-06-23T15:37:43.980

Thanks for all the attention.

I finally wrote to my hosting company and got a new set of IP's, now the attacks has completely stopped.

I'm still curios about how these attacks where done, so if anyone has any input - I'm still interested in a good answer. But for now the problem is solved for me.

Thanks again.

Bolli

Posted 2013-06-08T22:04:23.463

Reputation: 382

→ Bolli: do these SYN&ACK incoming paquets correspond to your SYN outgoing ones? If yes, analyse which process on your server is sending them. If, as I guessed it, they were not coming in reply to your real requests, these are just massive scans from a botnet. – dan – 2013-06-25T14:29:18.047