-2

The problem I am facing is that my Website (hosted on a dedicated VPS) was working fine for last 6 months. However from last two days, it gets huge spikes of DATA coming in and this causes server to crash. The hosting company shutdowns my server. I have tried bringing my website back multiple time but each time after some time same thing happens.

Regarding Dos Attack or other malicious tools, I would like to say that I am using:

Fail2Ban (for ssh and apache) Mod_Security (apache) Mod_Evasive (apache)

However there is not any solid ATTACK coming on the server.

The website is about a URL shortener.

**

  1. Details: OS: Centos 6.5 Processor:
  2. Intel Hexa-Core Xeon E5-2420
  3. Processor speed: 2GHZ
  4. Number of CPUs: 2
  5. Number of cores: 6
  6. RAM: 8GB
  7. Hard disks : 4 × 1TB

**

What can be the issue ?

Please guide me, I am a newbie. Have searched a lot of articles on INTERNET already but nothing is saving me at the moment ?

Update 1 Bandwidth usage image: http://oi57.tinypic.com/206fotk.jpg

Update 2 Also, the logs file generated by Apache server are to big e.g: 1. /etc/var/log/httpd/access_log (15GB) 2. /etc/var/log/httpd/error_log (5GB)

is this normal ?

Update 3 Formatted the complete Website. Reinstalled Ubuntu 12 LTS this... After 1 hour, the website went offline however this time there was no spike !!! I have contacted hosting company (THE LEASE WEB) but no response from them yet :-/

Arjun
  • 1
  • 3
  • Is your apache really crashing (see logfile and maybe oom messages in kernel log) or is you hosting company just shutting you vm down due to high load? Its a little unclear stated in your question. – Henrik Jun 04 '15 at 12:30
  • i think the hosting company shuts it down because of huge load (traffic spike coming in) let me attach a picture. – Arjun Jun 04 '15 at 13:43
  • 1
    The message from hosting company: The total amount of incoming traffic to the IP address *.*.*.* assigned to your account currently exceeds the limits of the interface (uplink port speed) assigned to you. This impacts the availability of your service(s) and the services of others. To prevent downtime and to minimize the impact on other services, we have null routed your IP until Wednesday 03/06/2015 at 11.00AM . We have done this in accordance with the Fair Use Policy and the terms of your contract. – Arjun Jun 04 '15 at 13:47
  • 2
    The message clearly states that it's the hosting company that is shutting your server down. You need to talk to them abou thtis. – Jenny D Jun 04 '15 at 13:48
  • 2
    I suggest switching the provider - 100mbit as spike usage are imho not a reason to emergency cut of a medium sized vm like yours. – Henrik Jun 04 '15 at 14:24
  • @Henrik but these spikes are not good. my website is average. i can't change the hosting company in the middle. :-( – Arjun Jun 04 '15 at 14:47
  • @jenny what should I say ? they are doing because its against the policy ? I am a newbie, I need your guidance ... moreover these spikes are not the normal. normal traffic is smooth, what do you say ? – Arjun Jun 04 '15 at 14:48
  • any specific place or LOG should I look for ? have tried read apache logs, ssh logs, fail2ban logs already but couldn't conclude anything ? :-( – Arjun Jun 04 '15 at 14:59
  • @Arjun tackling (D)DOS isn't easy at all - your mentioned steps (mod_security, mod_evasive, fail2ban...) are heading into the right direction but they need a lot of fine tuning. Regarding you provider: If this occurrence of traffic is against their policy you cant do much about this - maybe you want to switch to a dedicated box, with your own switchport... – Henrik Jun 04 '15 at 14:59
  • @Henrik thanks alot. but still confused... for last 6 months everything was fine :-( – Arjun Jun 04 '15 at 15:27
  • 1
    @Arjun Call your hosting provider and ask them for help in figuring this out and alleviating it. Your only other option would be to bring somo consultant in and actually pay them for help. This is not something that strangers on the internet can fix for you. – Jenny D Jun 04 '15 at 17:33
  • @JennyD Thanks ! have formatted my system and shifted to Ubuntu, now re-configuring everything from scratch ... will update you. – Arjun Jun 04 '15 at 18:22

1 Answers1

1

If it is a randomly generated IP DOS attack ,then your ISP is the only place it can be stopped.

However, if you can see the IPs are coming from a specific place, then read on.

Look at your /var/log/messages or syslog. Are your NICs going down? Bad Drivers - Are they Realtek?

If you want your server to be online while you figure this all out you can block the offending countries on your firewall. For instance, if you are fine with US and Canada traffic and the DOS/Spam isn't coming from the US or Canada, you can block the offending countries.

The Script:

# Create the IP block
ipset create geoblock hash:net

country=( br cn fr de hk in id jp ro ru tw tr kp kr )

for c in "${country[@]}"
do
        for IP in $(wget -O - http://www.ipdeny.com/ipblocks/data/countries/$c.$
        do
                # really big hammer - block country, period
                sudo ipset add geoblock $IP
        done
done

This blocks the geoblock ipset:

iptables -I INPUT  -m set --match-set geoblock src -p TCP --destination-port 80 -j REJECT

Add one for https as well, if you choose to.

Then, when you are satisfied you can try another test - this will open up your firewall again:

ipset destroy geoblock
Ehdrian
  • 78
  • 1
  • 5