9

Yesterday our Digital Ocean server encountered something that looked like an attack. The outbound traffic suddenly increased to 700Mbps, while the inbound traffic stayed at about 0.1Mbps, and didn't increase even once. The traffic lasted for several minutes until Digital Ocean cut our server off the network assuming we're performing a DoS (which is reasonable).

I have two assumptions: either someone hacked into our server (after the attack I realised my colleague had enabled SSH login with password) or there's some kind of an attack that I don't know about.

Can anyone clear this situation up for me? If there indeed is a kind of DoS which traffic looks like that, please educate me.

Krzysztof Kraszewski
  • 193
  • 1
  • 4

3 Answers3

20

One likely possibility is an amplification attack. If you are running an open recursive DNS resolver (there are other protocols you can do this with though), for example, you can receive a very small UDP packet that has a spoofed IP address. Your server then generates a large response and sends it to the victim, thinking that it's a legitimate request.

Another possibility is that someone was exfiltrating data off your network. If someone got into your server and was offloading every byte they could find, it would look like that as well.

There's no way to know which one it was without doing an investigation, and hoping that whatever did happen left evidence. If it's the latter (exfiltration) then they probably cleared their tracks as best they could.

Mark Henderson
  • 68,316
  • 31
  • 175
  • 255
  • 1
    Thanks. I'm in a corespondence with DO, hopefully they'll have an idea of what was going on. According to my investigation, it's likely that someone gained access to our server via SSH. I'm accepting your answer as it's the most precise in answering my question, although other answers are also very useful. – Krzysztof Kraszewski Apr 10 '18 at 16:24
  • 2
    @KrzysztofKraszewski Unless your colleague is/was using a really braindead password, SSH would NOT seem like a likely candidate to me. Remote brute-forcing is very slow and noisy. – Will Apr 11 '18 at 01:16
  • If the server was compromised, an amplification attack seems very unlikely. Why bother with such a trivial attack when you've rooted the server? And braindead passwords are remarkably common. – Phil Frost Apr 11 '18 at 02:20
  • 1
    @PhilFrost The point of me mentioning the amplification attack was that it's possible the OP is running something else that's just being used in that way and that the server has _not_ been compromised. DNS is the most common, but there's also MOTD and other weird old protocols that can be abused in this way. It is one possible solution that fits the weird traffic pattern. – Mark Henderson Apr 11 '18 at 02:21
  • 3
    [memcached is a particularly dramatic recent amplication attack](https://blog.cloudflare.com/memcrashed-major-amplification-attacks-from-port-11211/) – dave_thompson_085 Apr 11 '18 at 09:04
  • I agree an amplification attack is a good guess given the original question, but now that the OP has added in the comments "It looks like someone accessed our server and used it to perform an attack." I would guess it was _not_ an amplification attack. – Phil Frost Apr 11 '18 at 12:24
  • I would rule out an amplification attack (unless OP was running an improperly configured memcached instance) as the amplification factors for common protocols don't bridge the gap from 0.1Mbps to 700Mbps. Additionally, if they were exfiltrating data off your sever (rsync-ing or such), at 700Mbps-outgoing there'd be a significant protocol overhead that'd be registered as incoming traffic too, which I don't think (guesstimate here, math needed) 100Kbps is enough. Therefore I'd lean towards compromised node in a DoS botnet. – Nubarke Apr 11 '18 at 14:12
  • As it turns out, the reason was an exploit for software called VestaCP, which was installed on the server without my knowledge. See Savvior's comment under my question for details. Thanks for the answer anyway, I learned something thanks to all of you. – Krzysztof Kraszewski Apr 12 '18 at 08:48
10

I agree with the possibility of an amplification attack. The simplest way to handle this is to use DigitalOcean's free cloud firewall.

Only allow SSH, HTTP,and HTTPS inbound. If possible, only allow SSH from your trusted IPs.

You can do this using the firewall on your VM, DO's solution is just easier.

Mike M
  • 1,132
  • 4
  • 11
5

You should ask Digital Ocean. They don't shut off servers just for high outbound traffic: that would shut down most servers. For example, a webserver hosting something popular.

Rather, they shut down your server because the nature of your traffic looked malicious. As such, they probably have some idea what it was.

Otherwise you'll have to investigate yourself. Perhaps if the host is still running it's still attempting to send traffic which is being dropped by Digital Ocean. In that case you'd be able to observe it with a packet dump. Or you may be able to find clues in the system logs. It could be any of a million things unfortunately, so speculating on the underlying cause absent such an investigation is futile.

Phil Frost
  • 637
  • 5
  • 18