4

I recently signed up with Rackspace to host some database servers. I've got two MySQL servers set up, and have a method to create backups (using the Percona Xtrabackup and innobackupex tools). I've been trying to use duplicity to copy these backups to S3 and CloudFiles storage, and it is taking forreverr! I would expect the S3 backup to not be very fast, but the CloudFiles backup has taken 15 hours to backup 9GB. That's horrendously slow, and unacceptable for me.

I've looked through the duplicity source code, and by default it does not utilize the Rackspace Servicenet to transfer to cloudfiles. I then looked at the cloudfiles source code for the lib that duplicity uses for the CF backend, and saw that there was an environmental option for utilizing the Servicenet (RACKSPACE_SERVICENET). So long as that is set to something the cloudfiles lib should be connecting to cloudfiles via the Rackspace Servicenet, which SHOULD make for fast transfers. It does not.

I'm not sure if the speed limitation is because of some limitation of CloudFiles or if the cloudfiles python library isn't actually connecting via the RackSpace servicenet.

Do any of y'all have any other suggestions for how I should/could go about getting these backups off the server and onto a 3rd party or remote backup service?

Jim Rubenstein
  • 1,187
  • 1
  • 10
  • 16
  • InnoDB or MyISAM? – Michael Mior Jun 08 '11 at 13:53
  • I have both types, but the type is really irrelevant now, because I have a full backup of the database files. That part was taken care of by the Percona tools, I just have to get them off the server. I guess I should clarify the actual question...I started to lose my train of thought and stop paying attention towards the end. haha. – Jim Rubenstein Jun 08 '11 at 14:03
  • Are you sure that RackSpace is not throttling your bandwidth? Also, CPU scheduling may indirectly influence your transfer speed. I would eliminate these two factors first. – dtoubelis Jun 14 '11 at 16:40
  • I don't see why they'd throttle bandwidth on the ServiceNet to CloudFiles...I guess it's possible - not sure how to test that. As for CPU scheduling, the processors were under fairly heavy use - but I turned off encryption in Duplicity to reduce the overhead of creating the tar files which were transferred to CF/S3; other than that I'm not sure what all to investigate. – Jim Rubenstein Jun 14 '11 at 17:29

2 Answers2

2

Maybe not a full answer, more a suggestion. Could you not set up an Amazon EC2 instance that continually mirrored (or trailed by a few minutes) the main DB servers. Then you could run backups off that EC2 instance directly to S3 and get faster transfer speeds, as well as reducing the load on your primary DB machines.

Although 15 hours for 9GB is, if my mental maths is correct (which it probably isn't), less than 2MB/s which does sound like an issue. It might be worth contacting Rackspace support asking them why the slow transfer speeds.

Smudge
  • 24,039
  • 15
  • 57
  • 76
  • Running an EC2 instance that mirrors is an option I guess, but I question then becomes: what kind of mirror are you talking about? MySQL Replication is the most temperamental creature I've ever encountered, and restoring from backup once a week or once a month is just not a fun thing to do to get MySQL replication back online. Do you have an alternative idea to that? – Jim Rubenstein Jun 13 '11 at 12:19
  • @JimR I generally try to avoid MySQL at all costs =) That's what we do with our MongoDB servers (Yes I accept a completely different situation, but the theory might still apply). I'll be honest, I normally wouldn't answer a question like this but since it had no other answers I thought I'd throw it in anyway. Sorry I couldn't be of more help – Smudge Jun 13 '11 at 12:45
  • S'Alright (: I appreciate the effort, it at least gives me another idea to think about! – Jim Rubenstein Jun 13 '11 at 12:47
2

We use Rackspace Server Backup (a.k.a JungleDisk Server backup), which like Duplicity does local dedupe and compression and then uploads "chunks" via HTTP to a cloud provider. We saw some performance issues, and the underlying reason was our provisioning points for cloud files vs. cloud servers were different. Our cloud servers were being created in the DFW datacenter, but all cloud files buckets for JungleDisk are in the ORD datacenter.

Rackspace does not currently give people a choice for which datacenter they are going to use, because the DFW facility is near capacity. So everything for "newer" accounts as being provisioned in ORD. So you have to open a support ticket to get your provisioning point changed.

Also, you can't use ServiceNet between Rackspace datacenters (yet).

That said, we do see 40+ Mbps during backups even crossing Rackspace datacenters using Rackspace Cloud Backup, so I suspect you have some form of configuration issue with duplicity, or you are disk-or-CPU bound during the backup. Have you tried running the backup to the same target from outside cloud files? How does a simple HTTP PUT of a large file perform (i.e. exlude duplicity for a test)?

rmalayter
  • 3,744
  • 19
  • 27
  • Sorry for the delayyyyed comment, but I'll definitely test the HTTP PUT to cloud files and do some tests to see what uploading big files is like. I have a feeling that Duplicity (or the CF backend for it) is not actually connecting to the servicenet...maybe because of the cross-datacenter problem. Thanks again for the response, i'll report back with findings! – Jim Rubenstein Jul 20 '11 at 00:01
  • I signed up for Jungle Disk, and it works swimmingly. Eazy peazy to set up and get going, great backups, and *FAST*! – Jim Rubenstein Jul 29 '11 at 01:46