Office network HTTP traffic reporting tool

Question

At the office we'd like to have some sort of overview every month of the websites visited by the users in our network. Are there any existing tools running on Linux that can produce such a report by sniffing the traffic and generating URL statistics?

Answer 1

There are probably a number of ways to do this. One of the most common ways of doing this is to use Squid as a transparent proxy and then monitor the squid logs. Shorewall can help make it easier to set up.

You may find this post on monitoring office traffic helpful as well.

Answer 2

By sniffing the traffic????

In principle PasTmon can sort of do this - but a far better solution would be to force all traffic through a proxy like varnish or squid and analysing the logs.

Answer 3

There are pros and cons to both active and passive monitoring approaches.

As others have suggested, using a transparent proxy is one of the "active monitoring" options. It not only allows you to monitor but also to place controls once you have discovered errant usage.

On the flip side there are some cons if you are just interested in monitoring,

• It does take some setting up and administration both on proxy box and desktops
• Introduces a new entity into every connection (think point of failure)
• Cannot be used to account for non-HTTP traffic
• Not a passive solution

If you are already able to sniff the right traffic, for instance around your firewall out to the WAN, you have several passive options.

One new passive option is a Linux based solution called Trisul Network Metering and Forensics. It runs on an Ubuntu 10.04 or Centos 5.x 64-bit box. It will listen to sniffed traffic (or Netflow which is of course not relevant in this case) and produce all kinds of traffic reports. If you have the necessary paperwork in place in your country, it can also be used for recording all activity down to the flow and packet level and use that information to investigate security breaches.

For HTTP you can :

• Save all URLs requested
• Monitor top HTTP Hosts (from the Host headers rather than the DNS). This allows you to track sites like myblog.wordpress.com instead of just s2.wordpress.com.
• Top HTTP-Content types
• Top URL categories (news/radio/pron/etc)
• Other common stuff like Top Internal IP, Top External IPs, Apps, Subnets etc etc

I think ntop can also do most of the above too if you can configure it correctly with RRD or the MySQL plugin.

Trisul is totally free if you are monitoring a 3-day rolling window. In your case, you can generate and email daily PDF reports and then aggregate them on a monthly basis, so it wont cost you anything.

Answer 4

Loads if your using as a proxy server.

If you are using a domestic style BoradBand router most of these also have logging functions.

There are also potential legal problems with "spying" or going on "fishing Trips" in this way.

MK

Answer 5

ntop can do this, but if you don't already have a squid proxy you should get one set up - it'll make this sort of thing alot easier and give you the ability to filter what's allowed and what isn't, or throttle traffic.

Answer 6

As others have said, using a proper proxy and processing the logs from that is best. But for a cheap-and-cheerful fix, and assuming you've set up your switch infrastructure to sniff the traffic, I find that

ngrep -tiw -d eth0 GET port 80

filters out the URLs requested quite efficiently; sending that to a file then using awk or perl to post-process it, and extract only the pure URL, would be easy.

As others have said, you should be very sure that you have the consent of the owner of the network, and have at the very least informed all the users of this practice, before you do it.