3

I'm contemplating moving datacentre and I need to know the size of pipe I'll need in order to get a quote.

Currently I'm charged on GBs of traffic per month (currently about 42GB inbound - so requests only) but the new mob would rent me a pipe instead and the size of it determines the price.

I have a number of Centos web servers and one database server behind a loadbalancer; the webservers access the database server but the public do not (directly, that is).

The obvious place to measure bandwidth utilisation is at the loadbalancer but I don't have access to it and my current datacentre want an arm and a leg to monitor it for me.

Is this something I can easily do from my Linux web servers?

I've started to look at tools like ntop and bandwidthd but thought I'd seek some expert advice from here first.

I guess what I need to do is look at traffic between each webserver and the IP of the loadbalancer and add them together. Tools like ntop show traffic between the servers and the remote IP but not between the server and an intermediary IP...

Any clues anyone?

qryss
  • 133
  • 5
  • Do you have physical access & spare rack space? – Andrew Feb 02 '12 at 02:29
  • I do but I'd prefer to find my answer without needing that if at all possible. What were you going to propose, Andrew..? – qryss Feb 02 '12 at 03:16
  • 2
    A managed switch monitored via SNMP, using e.g. Cacti. – Andrew Feb 02 '12 at 03:26
  • If you are going the SNMP Route, there is MRTS, it does all the calculations for you and presents you with a neat table of traffic used. – Niko S P Feb 02 '12 at 07:10
  • 1
    My current datacentre want $600 to set up SNMP monitoring with $150 monthly charge thereafter for sending me the reports. Pirates! – qryss Feb 02 '12 at 09:38
  • you probably find that they don't really know what SNMP is and they can't afford paper for their printer...double PIRATES – The Unix Janitor Feb 02 '12 at 09:53
  • Just checking, but will you be moving all your servers into the new datacenter? Surely the new datacenter won't charge you for traffic between the servers inside the datacenter? – dunxd Feb 02 '12 at 12:55

4 Answers4

1

Do you really need an exact figure (whatever that means) to do this? Is knowing you currently use 100Mbps or 150.3528Mbps going to make a difference? Maybe you don't need to gather this data to get prices.

What pipe/price options do they offer? If they don't offer these, ask for quotes in bands that make sense to you. You may get a better deal by not revealing up front how much you actually need, and instead get them to reveal their pricing structure. And the info may be useful in future when you need to upgrade.

What does paying what you can afford get you? If it gets you way more than you expect to need now or in the next few months, then go down a level and ask again if you need that much pipe. Repeat till you are looking at a good price/pipe balance.

Then look at the options to upgrade. How long will it take to upgrade? Are their additional costs to change later? How much will it cost you to have too small a pipe for the time it takes to realise and implement change? That might direct you to go for a bigger pipe.

Answers to those questions will also be helpful in the happy circumstances of needing extra bandwidth because you are so successful in the future!

dunxd
  • 9,482
  • 21
  • 80
  • 117
  • [continuing from above] I've asked about most of this: with a 30Mbps pipe they come out a bit more than my current setup but they can size the pipe as finely as I want - and the pricing is (apparently) broadly linear (so half the size, half the price, etc). Following your suggestions I think I'll ask them what size pipe would make the cost the same as my current setup and see how that fits my estimates. It's not exactly the same offer, though, because I'd be moving from physical servers to a more actively managed virtual setup. – qryss Feb 02 '12 at 22:56
0

Inbound web traffic? Do you mean from the perspective of a browser or your server?

Your weblogs (hopefully apache?) should have the size of the content delivered out to users.

You can use those byte counts with timestamps to determine how much of your traffic is going out.

Be sure to convert from Bytes/s to Bits/s (8 bits in a byte).

Often pipes are billed at 95th percentile. (your top 5% of data is thrown out)

Some napkin math: 42GB/Month =~ 1.4GB/day =~ 0.02MB/s =~ .13Mb/s That's average. Your traffic peaks are likely 2-3X that. So a 1Mbps link might do the trick.

It depends on your traffic patterns.

As I said, you can get that via apache logs.

Joel K
  • 5,765
  • 2
  • 29
  • 34
  • Hi Joel, Thanks for that. By inbound I mean requests so the traffic out is the responses - definitely bigger than the requests but I don't yet know how much bigger. I'll see if I can get bandwidthd to give me those numbers - easier than trawling through logs (actually nginx). – qryss Feb 02 '12 at 09:29
  • Log parsing is a useful skill. There are also plenty of tools to help you do it. Knowing your traffic ratios is pretty fundamental to understanding your systems. Dig in to it. :) – Joel K Feb 02 '12 at 17:55
  • It's funny but I used to look at my logs massively; tbh I don't so much any more. Application logs - yes, all the time - but apache/nginx logs not so much exept when I'm solving some pernicious redirection problem or some such. I've taken to using Google Analytics for gauging site traffic (I know, I know) and the web servers have largely just worked for years without needing much study or attention. – qryss Feb 02 '12 at 23:04
0

you could run something like iptraf or just or reset your ifconfig counters. on each of your servers, take a sample over 24 hours to find out some average numbers. that will give you a ball park figure.

Have you thought about a CDN, which can offload a LOT of content delivery and load on your servers, cloudflare springs to mind. with a distributed CDN you could really cut down you server costs and bandwidth bills. Depends on the nature of your application and the data your pushing to your clients.

Oh yeah, i was in acronym mode CDN = content delivery network

The Unix Janitor
  • 2,388
  • 14
  • 13
  • interface stats will show both frontend (web) and backend (database) traffic, so it will be overcounted. – Joel K Feb 02 '12 at 17:55
  • Thanks for yr input. We're already using Rackspace Cloud Files as a CDN (been great so far - esp using the coudfiles gem 'cos we're running RoR). – qryss Feb 02 '12 at 22:58
0

You can use vnstat for monitoring your in & out traffic. This is a console-based network traffic monitor it's maintain the log of hour, daily & monthly also you can use vnstat php-frontend for graphical monitoring.

Configuration:

yum install vnstat

Once installed, you need to create a database with the following command:

vnstat -u -i eth0 ()

-u :forces a database update for interface or creates the database if it doesn’t exist
-i eth0 : use to specify interface

Please note that it will start to collect data via cronjob: 0-55/5 * * * * root /usr/bin/vnstat -u

for Vnstat PHP-Frontend: http://www.sqweek.com/sqweek/index.php?p=1

Community
  • 1
user103373
  • 186
  • 6
  • 19