5

When ever I try to load test a site that is load balanced on EC2 I get the following message:

ab -n 20 -c 20 http://www.somesite.com/

Benchmarking www.somesite.com (be patient)...Send request failed!
apr_socket_recv: Connection reset by peer (54)

Any sites that are hosted on EC2 without the load balancer can be load tested successfully, so I figure it's not related to the security group I have configured.

Does AWS EC2 load balancer block this type of request by default? If so how does it differentiate between an AB and normal browser based http request?

Steffen Opel
  • 5,560
  • 35
  • 55
Mitul
  • 139
  • 1
  • 8
  • Not as far as I know. Can you confirm that the ab traffic is *not* hitting the backend servers (via server logs or packet captures)? What is the result with a similar tool such as httperf? Perhaps ELB considers the ab HTTP request header invalid for some reason... – James Little Feb 01 '12 at 12:33
  • what are the results with "ab -n 1 -c 1", that is no concurrency? Does ab succeed in that case? If it does, you're likely seeing some connection-limiting behavior. – rmalayter Feb 01 '12 at 17:10

3 Answers3

6

Does AWS EC2 load balancer block this type of request by default?

Not that I know of - are you by chance running Mac OS X Lion? There seem to be a commonly encountered bug running ab on this particular operating system release - according to Fixing ApacheBench bug on Mac OS X Lion a patch is available (see there for details), though it's probably worthwhile and preferable to check for an official update first, insofar this patch (or an equivalent fix) has supposedly been incorporated in the meantime as of version httpd-2.3.15-beta at least (see Fixing Apachebench on OS X Lion).

Obviously, you could verify this first (and apply it as a workaround as well in case) by running the same test from a respectively different operating system (release).

Good luck!

Steffen Opel
  • 5,560
  • 35
  • 55
  • 1
    Thanks. Yep this is the exact issue, it's down to the current versions of ApacheBench in Lion. Can't confirm if patch works as haven't applied yet but will post if I get a resolution. – Mitul Feb 02 '12 at 13:22
3

Amazon ELB's are scaled up when load increases. When you have 5 requests per second you are on a different load balancer than the people running 1000 requests per second. What happens is that when you're running tests on the loadbalancer it goes from 5 to 1000 requests per second and the loadbalancer cannot handle the load so it will deny requests. If you wait 10-15 minutes for the loadbalancer to scale up while the test is running you should notice that all the requests will complete.

Amazon can manually upgrade your loadbalancer if you expect this is going to cause problems with your production environment. I have a loadbalancer / auto scaling group set up in the cloud and our traffic can go from a few hundred per second to a few thousand in a matter of 1-2 hours on the weekends and we don't actually see this behavior while running in the actual production environment, we only see this when we run tests with jmeter.

bwight
  • 793
  • 1
  • 6
  • 14
2

Load balancers are often configured to rate-limit or block requests that don't look like they came from "real browsers" as a security measure. I suspect that might be the case here; I'm sure Amazon has some docs.

Consider capturing a real request from your browser (using say Fiddler or Firebug) and add the headers you might need with ab's -H parameter. You often need a "Host", "Accept", and "Accept-Encoding" for example, perhaps with a user-agent string that looks realistic.

rmalayter
  • 3,744
  • 19
  • 27
  • Good point regarding load balancing in general, but it is possible to test ELB via `ab` just fine in principle, and the reported error would be different in case (likely _apr_socket_recv: Connection timed out (110)_), so I think this doesn't apply here - see e.g. [ELB capacity (simultaneous TCP sessions)?}(https://forums.aws.amazon.com/message.jspa?messageID=187513) for an illustration. – Steffen Opel Feb 01 '12 at 15:13