7

After searching and reading as many posts, comments and discussions I could find I did not find one specific to my issue.

I have multiple AWS EC2 deployments with a single RDS all in the same Delivery zone us-west-2(c)

I am testing the load of the instances at a fraction of what I will be expecting very soon. The issue that concerns me is the performance while pushing updates. We will be frequently acquiring updates of 1,000 records at a time and will take and compare our data and update our data as appropriate. Consequently one read and one write per entry. It is not uncommon to have 100,000 updates come to us in an hour.

Currently I have a MySQL Database on an AWS t2.medium class RDS running 5 update processes at 22% CPU and less than 1GB Memory.

Even with these low numbers the read time to search the database of 106.3K records takes 2 to 3 full seconds and the write time is another 2 seconds.

I need some thoughts on how to improve these read/write times.

Additional info: I have a replica instance running as well. The CMS driven sites (100 and growing daily) connect to the replica instance for their content.

Thanks!

Steven K7FAQ
  • 277
  • 2
  • 3
  • 12
  • If it takes 3 seconds to search the database I wonder if you've optimised your indexes and queries. A 2 second write time is also super slow, unless you have some insane triggers or indexes. T2 is burstable, it could be that you've run out of CPU credits and you're using only 10% of a core. Try a c4.large, it's twice the price but better suited for what you're doing. Maybe you could describe your application, sounds like a batch processing candidate. – Tim Apr 16 '16 at 09:04
  • @Tim, Thank you for the ideas. There are essentially two applications. 1) This issues application maintains up-to-date copy of real estate IDX records. Each time that a listing is updated we must also update our records. We maintain our own copy for numerous reasons. I am not sure exactly how batch processing would help in this instance but I will look into it. 2) The replica is for all the Internet visitors. That application is read only of the replica based upon their dynamic on-demand criteria. – Steven K7FAQ Apr 16 '16 at 14:50
  • I did upgrade the instance from a T2 Micro after seeing bottlenecks. I am thinking this is a tuning issue since I had better results on a single cpu 2GB VM node at a different provider. So far the CPU performance of this T2 Medium has not increased beyond 30% overnight. – Steven K7FAQ Apr 16 '16 at 14:57
  • 1
    Ok, interesting application. Remember with a t2.medium you only get 40% of a core, and once you run out of credits you're throttled, which is why a different instance type with constant CPU access could help. Strongly suggest you try the c4.large as a benchmark - spot instances are so cheap they essentially make testing free. – Tim Apr 16 '16 at 18:02

3 Answers3

6

I'm going to turn my comments above into an answer.

T2 instances only get a fraction of a CPU - 10% for a t2.micro, 40% for a t2.medium. You get CPU credits that build up, but once you use them you get throttled for CPU. You also get lower network and IO performance as you have to share physical machine resources.

What I suspect is happening is your testing has run out of CPU credits so you're being throttled. You can monitor CPU credits in CloudWatch, this will tell you whether or not my guess is correct.

Whether I'm right or not, t2 instances aren't suitable for systems that get work coming in constantly. A C4 or other general instance would probably be your best bet. You can monitor your database memory and CPU usage in CloudWatch to help work out what database instance size you need.

Tim
  • 30,383
  • 6
  • 47
  • 77
  • I upgraded to m4.large and still maxing out the CPU. Am going to look hard at optimization or another storage strategy. – Steven K7FAQ Apr 18 '16 at 19:21
  • Optimization is often very worthwhile, indexes and such are an easy place to start. m4.large is only two cores, for a busy database you may benefit from 4 or 8 cores. – Tim Apr 18 '16 at 20:22
4

Take note of the disk bottleneck : Disk IOPS. RDS will throttle IO request if it is not provisioned. There is 3iops/GB given to SSD. So if you have 100GB allocated for SSD, you can max up 3x100 = 300 iops block.
If you pay for provisioned SSD, you can provision iops more than that.

While for standard(magentic) storage, additional per millions iops charge count, and the burst is 40-200 iops.

Please go check your RDS io log on the activities to determine better disk io strategy. More info reference : https://stackoverflow.com/questions/18777070/aws-rds-provisioned-iops-really-worth-it

Community
  • 1
mootmoot
  • 294
  • 1
  • 6
1

I ended up creating a c4.xlarge EC2 and having it operating as my MariaDB 10 server. Due to the large amount of traffic currently and a major increase coming with new marketing campaigns I could not depend on RDS without spending several hundreds per month. And I found numerous complaints about AWS RDS with similar issues as mine.

Now the system is able to maintain two update workers simultaneously while visitors are able to get the dynamic content without a decrease in performance.

Thank you everyone for your thoughts and comments!

  • Steven
Steven K7FAQ
  • 277
  • 2
  • 3
  • 12
  • Interesting. Did you work out the bottleneck, CPU or IO? Did you look at IO on your RDS instance? Limited IOPS could be the issue. RDS is meant to be very scalable. Did you need to change your application? – Tim May 08 '16 at 21:03