How to determine what's causing high bandwidth usage on server

Question

I have a MySQL database server running on a VPS. The server is connected to a web server via a private network interface, and has no public services except SSH and a PHPMyAdmin installation.

Some time ago, I installed a Monit rule to check for abnormal traffic over the Public network interface (AKA, not the one that MySQL data goes over):

# Monitor network connection
check network public with interface eth1
   if failed link then alert
   if changed link then alert
   if saturation > 90% then alert
   if download > 10 MB/s then alert
   if total upload > 1 GB in last hour then alert

A few months ago, I started getting flurries of Monit alerts triggered by this rule:

Upload bytes exceeded Service public

        Date:        Tue, 04 Apr 2017 13:11:55
        Action:      alert
        Host:        myhostname.com
        Description: total upload 1.0 GB matches limit [upload rate < 1024.0 MB in last 1 hour]

Your faithful employee,
Monit

These alerts don't really correlate with any spike of usage on the server, and since there are no real public-facing services I can't imagine what could be responsible for this amount of upload bandwidth. Disk space doesn't seem to be being used either.

How can I find out what is causing 1GB+ of "something" to be uploaded, what is being uploaded, and from where?

If MySQL should only be used over the private interface, you could block port 3306 on the public interface in your firewall rules. — Barmar, Apr 04 '17 at 20:55
I control the server, and 3306 (as well as most other ports) are blocked on the public interface. — MarathonStudios, Apr 05 '17 at 02:16

score 1 · Answer 1 · answered Apr 04 '17 at 19:48

Sounds like something is running some kind of periodic SELECT query that's dumping data out the public interface.

Wireshark, pcap, or other variant of packet snoop would reveal source (the SQL server) and target, ports/protocols used, and so forth. Zero in on port 3306 (either TCP or UDP), which is the MySQL port.

If you've a lot of disk space on the system you can set your packet snoop up to save the pcap files containing the network traffic over your public interface.

I would recommend spooling mode, say, 10GB, broken into 2GB files, such that you never use more that that amount of space, but you should be able to capture at least one of these uploads (or at least the start of it). It may take an hour or two to fill that, or only a few minutes, but you'll always have at least 8GB of intact conversation using this method (the last file will be <2GB and will be the newest data).

Once you have an idea of what's happening, you can take steps to mitigate. Right off the bat, I'd recommend closing port 3306 on the public port and also locking down "May run queries" creds within MySQL's permissions. That'll lock the barn door.

score 0 · Answer 2 · answered Apr 05 '17 at 15:56

A possible a way to identify what is happening from the MySQL side is to enable the slow query log then establish a really low threshold to capture every statement. Then you can look through the slow query log to see what was running at that time.

An alternative approach would be to dump "show full processlist \G" periodically. This will be much easier to see what is running at a given time.

How to determine what's causing high bandwidth usage on server

2 Answers2