5

We recently moved our web servers from some 10 year old boxes to AWS EC2.

Usage of the site is currently higher now (it's our busy season) and the site has become much slower, which is unexpected because our instance sizes are much higher than what we had previously.

We run a pretty small site that only gets a few hundred at a time. We're running a c3.large instance on our webserver, and db.m1.large for our RDS MySQL database. We don't have any read replicas and or multiple webservers (load balancing). According to Google Analytics, we only had 18,106 page views for the whole day.

Our users (external and internal) keep seeing the site time out in their browser. It's pretty much across the board instead of any particular page. MySQL PROCESS LIST is also nearly empty without any table locks or whatnot.

If you look at our stats in CloudWatch, everything should be fine. We have very CPU utilization, and what I think is pretty low Network I/O. Likewise on the RDS side, nothing here is screaming "bottleneck".

EC2 Usage (c3.large) UE2 Usage

RDS Usage (db.m1.large) RDS Usage

Any ideas how I should go about troubleshooting this issue?

DOOManiac
  • 781
  • 6
  • 12
  • 25
  • RDS? As in Windows Server? Have you tried using PerfMon to see what the OS sees? Also look at the SQL stuff too, it may be stuck running a query. – MDMoore313 Feb 25 '14 at 18:51
  • 1
    18,106 pageviews a day means one every 5 seconds on average, yet you're averaging 50-100 writes and 200-400 reads per second. I'd troubleshoot that first. – ceejayoz Feb 25 '14 at 19:13
  • 2
    @MDMoore313 - no RDS as in AWS's managed db service. – EEAA Feb 25 '14 at 19:42
  • @EEAA In fairness, RDS offers Windows SQL Server. OP said MySQL somewhere in the post, though. – ceejayoz Feb 25 '14 at 19:43
  • Sorry, forgot to add: EC2 instance is Amazon Linux w/ Apache/PHP/Memcached (although we just started using Memcached and only have it on 2 or 3 places so far) – DOOManiac Feb 25 '14 at 19:56
  • @DOOManiac it seems that your EC2 instance is using EBS, but it seems that the statistics of that is missing. Could you post that as well? – Raymond Tau Feb 25 '14 at 21:13
  • Yes, it is an EBS backed instance. – DOOManiac Feb 25 '14 at 21:43
  • Are the RDS and EC2 instances in the same availability zone? – David Levesque Feb 26 '14 at 02:02

1 Answers1

6

Finally tracked down the cause of our problems. Apache was misconfigured to have a Keep Alive timeout for 30 seconds. That combined with an overly aggressive AJAX script was causing the site to hang and wait for a connection.

Turning KeepAliveTimeout down to 7, as well as taming the AJAX script, brought everything back to normal.

DOOManiac
  • 781
  • 6
  • 12
  • 25