9

I have a virtual server running Ubuntu 18.04 from a well known hosting company. This morning our Fortigate Firewall logs shows that my Win10 computer transferred 3.5TB to and 6.5TB from my virtual server over 13 hours (over last night) via SSH.

There are a couple of issues with this; First we know the figures quoted by the Fortigate Firewall are not correct, because a) the connection speed isn't fast enough able to do this over that time period, perhaps a 10th of the required speed, and b) the VPS logs show that it received 35GB and sent 65GB... many times less than the Fortigate reports. And secondly the only things that were open using SSH were Putty and WinSCP. Putty wasn't doing anything, and the bash history shows the only commands were related to starting and stopping Kestrel (dot net core server), it was doing nothing else. WinSCP wasn't doing anything either as far as I can tell. I'd moved a few folders around during the day.

Nothing came up with a virus scan on my local machine, there wasn't anything awry in the server logs, and the server has no third party packages other than the Microsoft .net core repos set up.

While the boss is OK that nothing sinister was going on I am not happy that something has ocurred that I can't even begin to answer.

I have a whole host of questions about this but right now I'll settle for:

  • What could have caused this?
  • Could it just have been an issue with the Fortigate as we know it's mis-reported the data transfer, or perhaps WinSCP in a loop?
  • Has anyone ever had this happen to them at all?

Any clues gratefully received.

CodePoint
  • 154
  • 6
  • 4
    Check with your VPS provider and see if they can verify the amount of traffic from your IP address as a sanity check. – Tim Brigham Jun 03 '19 at 13:18
  • Yes, I have confirmed that the report is correct. The whole thing is a big mystery. – CodePoint Jun 03 '19 at 14:14
  • @CodePoint The numbers seem to match, even if there's a 100x difference? If that's correct then maybe you could try uploading say 1GB or 10GB to the server, through ssh connection and then check what the firewall logs show. Maybe it constantly misreports the transfer but under normal circumstances it doesn't stand out? Then you would "only" need to figure out what caused the transfer. – Tomasz Zieliński Jun 03 '19 at 14:39
  • @TomaszZieliński I won't get permission to do that. However my boss is sufficiently satisfied that the firewall reports are wrong, and he's now going to raise this with Fortinet to advise, and I suppose we'll see what comes back from that. Now I'm just wondering what on earth it was that was being transferred, because either way the unknown 35GB/65GB transfer is still not acceptable, at least to me. And I hate to leave this as 'just one of those things'. – CodePoint Jun 03 '19 at 14:47
  • 3
    If you don't have authority to investigate the firewall, then there's very little more you can do at this point, but pass it on to whoever does have that authority. – Michael Hampton Jun 03 '19 at 14:52
  • @MichaelHampton Thanks. I think you're probably right. There is obviously something not right with the firewall with reports that far off, so I assume it is possible that the problem might only exist between the firewall and the server. That might also explain why I cannot see anything on my machine that explains the transfer. Is it possible that a firewall can get stuck in sending/receiving loop? – CodePoint Jun 03 '19 at 14:59
  • 2
    @CodePoint One of my (very) wild guesses was that something was sending some data that compressed very well, and the firewall reported the size before compression. Now, the numbers are too perfect for that and also it would probably have to be your firewall sending the data (unless it's set up to MITM SSH?), but I'm writing it just in case.. – Tomasz Zieliński Jun 03 '19 at 15:37
  • @TomaszZieliński The firewall is certainly set up to MITM SSL web traffc, so I assume it would extend that to SSH too perhaps? – CodePoint Jun 03 '19 at 15:42
  • 2
    @CodePoint A quick search shows that Fortinet can do that, but that's beyond my expertise, so to speak. But even in case even if it can't MITM SSH (or is not configured to do so) it can send a lot of data to your VPS if only there's a bug or misconfiguration (and things happen..). But I admit it's just a total speculation on my part, and as Michael Hampton wrote above someone needs to investigate the firewall to say anything for sure.. – Tomasz Zieliński Jun 03 '19 at 16:11
  • 2
    @TomaszZieliński Thank you for your input. At least I've got some where to start looking now. – CodePoint Jun 03 '19 at 16:13
  • How are you getting the Fortinet numbers? I've many people forget to divide *bytes* down to get to megabytes or gigabytes (Fortinet syslog, iirc, reports transfers in bytes - so if you're not dividing, and instead thinking you're seeing megabytes or gigabytes, you'd be unnecessarily alarmed; also, iirc, some firewalls report the same bytes in multiple ways - so if you're not filtering on that, too, you could be multiplicounting the same data). – warren Jun 03 '19 at 18:14
  • We did look at that in the first instance, but it most definately show the data in TB. – CodePoint Jun 04 '19 at 07:59
  • To transfer 6.5Tbyte of data in 10h, you'd need more than 2gbps of bandwith. Are you sure about thse numbers? – bjoster Nov 11 '19 at 14:14
  • Very sure that these are the numbers that Fortigate displayed. Also very sure that our connection could not possibly handle that. This is essentially what my question is about. Fortigate claimed that's what was transferred, yet we know it can't possibly be true. – CodePoint Nov 26 '19 at 16:01

4 Answers4

1

I would start by changing all the passwords and enforce longer and more complicated passwords. You may have an account that is compromised. It happened to me and I decided to change passwords, reboot, and that stopped the hacking dead.

I was lucky, I guess, the hacker did not implant something to gain access again.

Bernard
  • 11
  • 1
1

Check processes running on your ubuntu or windows machine.

There might be some spam planted by some hacker, which is running in background and causing network flooding.

saurabh
  • 56
  • 2
1

If server is linux and the size of traffic is within the size of the server then maybe someone cloned the server? VCenter Converter can just use ssh and of course a root access to do it.

Kenkoy
  • 11
  • 2
0

Something is flooding your network, capture a package with Wireshark to perform network traffic analysis.

Keftef
  • 86
  • 1
  • 8