4

I'm in the process of moving a webapp from the local server that's been its development home to somewhere else for eventual deployment. I'm currently evaluating AWS and Rackspace, and am in the middle of trying the site out on AWS. I've tried to keep the two environments as similar as possible; both are running a pretty straightforward LAMP stack on top of Fedora 17, with the same versions of apache, php, and such. My development machine is a homebrew box based on an i7 860 chip with 32 GB of memory; at AWS, I'm typically using an m1.small instance, built from their "standard" fedora 17 instance, which gets described as "Intel Xeon E5-2650 0 @ 1.80GHz (1 Core), Memory: 2048MB" by some benchmarking software I've been using. The root device of my AWS instance is set up as an EBS volume.

The site is up and running on both boxes, and I was pleased to see that site performance is roughly comparable, with AWS being a little slower. However, I'm also doing some video encoding as part of the site's work, via a version of ffmpeg I've built from source on both sites. Here, I'm getting a huge performance difference, with my development server being about a factor of 10 faster than AWS. I've run a few benchmarks, and they show a similar difference: the Phoronix "apache" benchmark shows my server running about 12x the AWS instance.

So, I'm puzzled. I understand that the "E5-2650" description of the AWS instance is just for descriptive purposes, and that I don't really have a machine with an E5-2650 all to myself. But what's the right way to think about this? The E5-2650 seems to be a crazy-fast 8-core chip, by some measures about twice as fast as my i7; maybe I should be thinking that I effectively have 1/8 of such a machine (1 core out of 8)? That still doesn't get me to 10x, but maybe that's attributable to the (much) larger amount of memory of my development machine? Or have I screwed up something with my AWS installation -- I'm about one step above a complete AWS newbie, but not much more, so screwing something up is quite possible. Any advice out there?

Jim Miller
  • 713
  • 2
  • 11
  • 23

1 Answers1

3

I have a few options for you to consider:

Option 1: IO Contention

The disparity in your video encoding performance likely has very little to do with CPU. Rather, look into IO contention. While you're encoding, run top, and watch the value of %wa in the header. This is the percentage of time the server is having to wait for IO requests. You want this number as low as possible. On my systems, I start making changes if I see this number get any higher than 5% or so, averaged out over a 5-minute span.

If you are indeed seeing high IO contention, there are a few things you can do. First, and perhaps easiest, would be to move to a Provisioned IOPs (PIOPs) EBS volume. With these, you can specify how many IOPs you need. You'll pay extra for it, but it's by far the easiest way to increase IO performance.

If you don't want to do that for some reason, you can associate a bunch of EBS volumes to your server and stripe them together into a larger RAID0 volume. Aggregate IO will be greatly increased in this situation, but it's also much more risky, as the failure of any single EBS volume will destroy the data on that volume.

I'd recommend giving a PIOPs volume a try.

Option 2: CPU Throttling

Another thing to consider: you're running an m1.small. On these smaller instances, AWS is pretty aggresive with CPU throttling, so you may consider moving to a larger instance, possibly a High-CPU model, like the c1.medium. If AWS is throttling your CPU, you'll see high values in the %st (steal) column in the output of top.

Option 3: Outsource it

This may be a stretch, but why not try out AWS's Elastic Transcoder service? That will take all of the burden of encoding off of your server completely.

EEAA
  • 108,414
  • 18
  • 172
  • 242
  • Very helpful; thanks! I tried an ffmpeg run while watching top, and, with my m1.small instance, %wa stays pretty close to zero, but %st bounces between 40% and 60% when ffmpeg starts cranking (and using up 9x% of the server. On my development server, %st never budges. Any suggestions? BTW, I am separately looking into the Transcoder service -- maybe the money I spend on the transcoder service will keep me from having to provision more powerful servers... – Jim Miller Jun 21 '13 at 01:10
  • Well your dev server isn't subject to CPU throttling, so the fact that you aren't seeing any CPU steal shouldn't be surprising. There's nothing you can do except upgrade to an instance with more CPU. – EEAA Jun 21 '13 at 01:40
  • BTW, I ran the site as an m1.medium, just for grins. I don't have "top" data on this right now, but the ffmpeg processes ran about twice as fast as with m1.small, as you'd expect. If I get around to trying the "top" test, I'll report back. – Jim Miller Jun 21 '13 at 02:15
  • You can also use `iostat` to see if you're going over the 100 IOPS you get from ELB by default. – David Schwartz Jun 21 '13 at 03:09
  • Very good answer, thanks! As a Amazon newbie, I really love to read these. – tomi Jun 21 '13 at 08:11
  • BTW #2: I rebooted the site as an m1.medium. My ffpmeg job ran with 0% %st, but still only ran about twice as fast as the m1.small instance. The Phoronix "apache" benchmark also ran with about 0% %st, but about 12% %si (software interrupts) and also came in at about twice the speed of the m1.small. – Jim Miller Jun 21 '13 at 16:03
  • Final BTW, I promise: I tried the site as a "compute-intensive" c1.medium instance, and got much better results: no throttling, and both ffmpeg and the apache benchmark ran twice as fast as with the m1.medium instance. So I'm within a factor of 3 of my dedicated development machine, which is seeming much more reasonable. – Jim Miller Jun 21 '13 at 16:32