1

Over the last day or so, one of instances has had a large amount of bandwidth consumed. This means that we have nearly exceeded our allowance (roughly the same amount inbound and outbound).

Looking though the logs, the only thing I can see is lots and lots of 400 172 errors in the nginx access.log with the same text string.

I have changed nginx to a different port, implemented fail2ban but since the traffic is coming from different IPs this isn't working. I have also got our VPS provider to change our VPS' IP.

Fail2ban is currently dropping all connections to Port 80 which isn't ideal as we'd like to use this port.

Is there anything we can do to improve the situation? If we are dropping suspect traffic will this still count towards our allowance?

More Info

I managed to get more details by changing the nginx error log level.

The only error that seems to be occurring is cleint sent an invalid request while reading client request line.

The domain is new and has not being used before ( it is a brand new subdomain on one of long term existing domains).

I wil check whether it is using the same path.

Also is there any reason why its increasing outboubd traffic is it just because the inbound packets are being blocked?

user2099762
  • 133
  • 2
  • 4
  • 18

2 Answers2

3

In general, any traffic that reaches you or that reaches the gateway (when outbound from your host) counts against your bandwidth allowance. This is true even if the activity is insubstantial or consists only of traffic blocked by your host firewall.

Dropping the malicious traffic as soon as possible is ideal. However, it sounds like you might have a problem involving a hosting plan that has large bandwidth but small transfer quotas - you are experiencing a fundamental problem with such plans, if this is the case.

What you do from here depends on what the traffic is like. If it is only a few hosts, consider blocking those hosts using the VPS provider's firewall, if they make this option available. If it is a search engine robot, add a robots.txt prohibiting them from crawling whatever path is generating the errors.

172 in your logs isn't part of the error code - these lines only mean your server returned the error code 400 Bad Request and the error page it sent back was 172 bytes in length.

Since it is error 400, it's most likely not a search engine (unless you are running a strange CGI application), but without knowing what the query string (including method) was, it's hard to say what is happening. But, try addressing it based on what it looks like the "attacker" is trying to do.

Blocking the requests in nginx is unlikely to change much.

Instead, you can consider blocking these requests at the network layer by dropping offending traffic; this means the originator of the traffic will end up with a bunch of open connections. If the velocity does not change, odds are fairly good it is malicious. You might consider sending an abuse complaint to the originating ISP if it is all coming from the same network and it looks like a flood rather than an automated process trying to do something. It might be as simple as someone else having had your domain name before you.

Falcon Momot
  • 24,975
  • 13
  • 61
  • 92
1

Without knowing about your hosting provider I can't give a definitive answer regarding whether the denied requests count towards your allowance but in general if it reaches your server instance it is usually counted.

Using fail2ban is definitely recommended in this circumstance. Ensuring you are dropping the inbound packets and not denying them will mean you shouldn't see any increase in outbound traffic in the form of 'Connection refused' ICMP packets.

Is there any pattern in the incoming requests? Are they all from a specific CIDR block or are the requests for a specific URL? Is this a URL something legitimate on your website or a random string?

  • If it's the same URL every time I'd suggest blocking those requests in Nginx if you haven't already.

  • If it's a specific source CIDR block which is reasonably small you could just blanket drop all connections from those addresses.

It sounds like you are being DDoS'd. This Server Fault article provides excellent advice: I am under DDoS. What can I do?

Community
  • 1
Ben Fairless
  • 76
  • 4