1

My VMWare ESXi 4 server appears to be under a Denial of Service attack. I am getting massive packet loss to the server (60+%) and am barely able to load any services on the VMs running on the host.

I have Cacti installed but cannot load it due to the attack. I can SSH in to the VMware host. Are there any commends I can run to either determine where the attack is coming from, or block all IP addresses except mine so that I can load Cacti again to troubleshoot?

I tried esxcli network firewall get but received: Unknown Object firewall in namespace network

All the VMs with network access are directly connected to the internet, that is, there is a virtual switch between the internet-facing VMs and the router.

EDIT: MDMarra had a great idea: disable the vswitch that the VMs are on. But I can't get the vSphere console to respond long enough to do this. Can this be done through SSH?

Josh
  • 9,001
  • 27
  • 78
  • 124
  • You _have_ called your ISP, right? – Michael Hampton Jul 27 '12 at 19:34
  • Yes, thanks @MichaelHampton. They told me it was a DoS / DDoS but were unable to assist beyond that. If I can't get this resolved soon I am calling them *again*. I was hoping maybe SF would be more helpful, because they were not! – Josh Jul 27 '12 at 19:36

3 Answers3

2

I would say first and foremost would be to call your datacenter and see if they can block the offending IP with their equipment. Hopefully their hardware has the bandwidth to handle something like that, which will then at least allow yours to start functioning like normal.

Safado
  • 4,726
  • 7
  • 35
  • 53
  • 1
    Thanks. Unfortunately they were unable to even tell me where the attack was coming from, much less block it, which I found patently absurd! – Josh Jul 27 '12 at 19:31
  • Wow, that is unfortunate! – Safado Jul 27 '12 at 19:43
  • Yeah, it's because they want to sell me a $500/mo DDoS prevention plan rather than helping a long time customer. It's time for a new ISP. – Josh Jul 27 '12 at 19:48
  • @Josh, do you have console access to this machine? If not, you should make plans to do so going forward. – Mike Pennington Jul 27 '12 at 19:54
  • Thanks Mike. I did when I performed the initial setup through a IP KVM, but I stopped that service after the install because I was able to do all I needed to through SSH / vSphere center... – Josh Jul 27 '12 at 19:55
  • @Josh, until now ;-)... and thus worth reasonable monthly charges, even if that means it's sitting idle 350 days a year on an OOB IP address – Mike Pennington Jul 27 '12 at 20:10
  • You can do things via OOB console you can't do via SSH now, correct? OOB management is part of good sys admin best practices. I think you just told Paul Ackerman that you can't run tcpdump; guess what, if you paid for OOB access, you could. If you don't hook up OOB, that's understandable, but also know that you *are* gambling if you choose no OOB management. – Mike Pennington Jul 27 '12 at 20:38
  • @MikePennington Sadly, my ISP's pricing isn't reasonable. I could have them add an IP KVM right now, but what good would that do me? Can I do anything from the console I can't do via SSH – Josh Jul 27 '12 at 20:38
  • You could stay connected for starters which I imagine is difficult to do with 60+% loss. – Paul Ackerman Jul 28 '12 at 01:04
1

The ISP was unable to determine the cause of the traffic, but what they were able to do was null-route all the IP addresses assigned to this server at the network switch. Then, one-by-one we removed the null routes, until we determined which IP addresses were being attacked. Once the target IPs were null-routed, the problem went away and I am able to access the server again.

I am now going to console in to the affected VMs and start tcpdump, and then remove the null-routes to those VMs. This will allow me to find the source IPs of the attack, which can be blocked by my ISP before traffic from them enters the core network.

Josh
  • 9,001
  • 27
  • 78
  • 124
0

Sniff the wire and filter traffic to just that host. tcpdump / wireshark

Paul Ackerman
  • 2,729
  • 15
  • 23
  • There is no tcpdump on vmware ESXi, and I am getting so much packet loss that when I try to run it on the virtual machines, it just locks up. – Josh Jul 27 '12 at 19:23
  • 1
    you misunderstand. Configure a span port on your switch so you can monitor the traffic with a laptop. Span whatever Vlan your server is in. – Paul Ackerman Jul 27 '12 at 19:24
  • Sorry for not being clear, I'm in the middle of trying to fix this ASAP as all my customers' sites are down. This is a managed server, I have no access to the switch. I will ask the datacenter if they can do that, however, thanks – Josh Jul 27 '12 at 19:28
  • ahh. I see. In that case, can you run vm-support and get VMware to help you diagnose? – Paul Ackerman Jul 28 '12 at 01:09
  • I don't know the specifics but the vm-support utility can diagnose various performance issues so it may provide insight into the network traffic. – Paul Ackerman Jul 28 '12 at 01:36