27

I have Apache set up to serve several Virtual Hosts, and I would like to see how much bandwidth each site uses. I can see how much the entire server uses, but I would like more detailed reports.

Most of the things I have found out there are for limiting bandwidth to virtual hosts, but I don't want to do that; I just want to see which sites are using how much bandwidth.

This isn't for billing purposes, just for information.

Is there an apache module I should use? Or is there some other way to do this?

pkaeding
  • 790
  • 2
  • 12
  • 23

8 Answers8

25

The information you're after is all in the logs, so you should look at a log analyzer such as AWStats. The other option is to use Google Analytics.

For analyzing the logs, here's a rough example which you can use to tell you how many MB of traffic a log file reports from the command line:

cat /var/log/apache/access.log | awk '{SUM+=$10}END{print SUM/1024/1024}'
Giacomo1968
  • 3,522
  • 25
  • 38
Xerxes
  • 4,133
  • 3
  • 26
  • 33
  • 5
    useless use of cat: awk '...' /var/log/apache/access.log works too – marco Jan 12 '11 at 17:43
  • Will the suggested way(s) calculate the traffic sent from some web page(s) to the Internet (outbound traffic)? – Khaled Jan 13 '11 at 07:36
  • 2
    Yes. But be careful that you dont sum up the response codes by accident. I realised my counts were way too low because for me $10 is the http response code. For my apache custom log I needed to use {SUM+=$11} instead. – Phil Jan 20 '15 at 17:17
5

I suggest you use the wonderful apache logging mechanism and its less known %I and %O flags:

Define the format:

LogFormat "%t %a %v %U%q %I %O" IOFormat

Use it in your main httpd.conf:

CustomLog /var/log/apache2/all-bw.log IOFormat

The values are probably not accounting all headers information, but are quite accurate to have a precise idea of VirtualHost traffic.

Scan the logs with a perl script to aggregate per virtual host every n minutes (5 for example) and send this to cacti.

These flags are provided by mod_logio which is probably built in your Apache (as it for my Debian's Apache).

Oct
  • 1,576
  • 1
  • 8
  • 5
  • 2
    Hum, also as quoted in Apache 2.0 documentation: Note that in httpd 2.0, unlike 1.3, the %b and %B format strings do not represent the number of bytes sent to the client, but simply the size in bytes of the HTTP response (which will differ, for instance, if the connection is aborted, or if SSL is used). The %O format provided by mod_logio will log the actual number of bytes sent over the network. – Oct May 29 '09 at 16:24
  • As of apache 2.4.7 there is now `%S` which is total bytes transferred, sum of both `%I` and `%O` https://httpd.apache.org/docs/trunk/mod/mod_logio.html – Wadih M. Jul 25 '22 at 01:05
3

Awstats is one way to do this but probably not the best

Antoine Benkemoun
  • 7,314
  • 3
  • 41
  • 60
2

If you decide to use awstats with Apache, out of the box it will show you aggregated bandwidth for your entire server.

To see bandwidth on a per virtual host basis, I recommend installing vlogger.

Vlogger will actually gather Apache access log information for each of your virtual hosts that you set up to do so in separate directories/files.

For example if your Apache log file is in /var/log/apache2, typical vlogger installation will create something like this for your virtual hosts (e.g. vhost1.com vhost2.com): 

/var/log/apache2/vhost1.com/access.log
/var/log/apache2/vhost2.com/access.log

Vlogger gives you the option to rotate these logs for you, provides a way to change the naming template of the access log file (e.g. add a date), and claims it handles a large number of log files better than Apache.

One down side to this is that you won't have an aggregated server view anymore (you'll need to aggregate logs separately or perhaps use an additional apache setting or perhaps some other method?).

I would caution against using google analytics (or any javascript based tracking) for server bandwidth monitoring as you are relying on the client to report via the javascript. GA does not report to you people who have their javascript disabled as well as any crawlers/spiders/bots.

user12345
  • 173
  • 1
  • 6
1

Here is some regex to parse the log format proposed by Xerxes.

\[([0-9]+)/(\w+)/([0-9]{4})[^\]]+\]\s(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\s([^/]+)[^\s]+\s(\d+)\s(\d+)

Example log:

[12/Jan/2011:14:25:04 +0000] 157.157.12.206 files.hjaltijakobsson.com / 581 669 [12/Jan/2011:14:25:04 +0000] 157.157.12.206 files.hjaltijakobsson.com / 624 747 [12/Jan/2011:14:25:04 +0000] 157.157.12.206 files.hjaltijakobsson.com /icons/blank.gif 687 186 [12/Jan/2011:14:25:04 +0000] 157.157.12.206 files.hjaltijakobsson.com /icons/compressed.gif 693 188 [12/Jan/2011:14:25:04 +0000] 157.157.12.206 files.hjaltijakobsson.com /favicon.ico 592 512

Matches:

Subpattern 1 (day of month): 12
Subpattern 2 (abbr. month): Jan
Subpattern 3 (year): 2011
Subpattern 4 (visitor host): 157.157.12.206
Subpattern 5 (virtual host): files.hjaltijakobsson.com
Subpattern 6 (incoming bytes): 581
Subpattern 7 (outgoing bytes): 669

Cheers.

hjaltij
  • 111
  • 2
1

Slight tweak on the accepted answer assuming there is actually multiple vhosts on the server (and therefore multiple site.com.access_log 's). This will sort and list each vhost 

for f in *.access_log ; do echo `awk '{SUM+=$10}END{print SUM/1024/1024}' $f` $f ; done | sort -rn | more

and for a directory of gzipped logs

for f in *.access_log.*.gz ; do echo `zcat $f|awk '{SUM+=$10}END{print SUM/1024/1024}'` $f `stat -c%z $f`; done | sort -rn | more
michael
  • 111
  • 2
0

Hmm, you could get evil with IPTables and string matching to log the packets for later reporting. Will only work for non SSL connections though.

Or something protocol and session aware like Snort could be shoe horned into use ...

Rob Dudley
  • 51
  • 6
0

Correct. Filtering the log is a good idea. I also want to get the bandwidth of my Apache server when download files.

LogFormat "%h %l %t %u \"%r\" %>s %b/%D**" combined

Calculates the %b and %d output, which will give you the bandwidth of current.

slm
  • 7,355
  • 16
  • 54
  • 72
David
  • 101
  • 1