3

We're not very pleased with our current colocation provider and so we're looking to move to a different company in a different city. Our business is an online school that's accessed world wide (which means we need 24x7 availability to our application, but maybe 1-2 hours max of downtime is affordable). Colleges and Universities around the world offer our curriculum as their own, so part of those agreements say we can have no more than X amount of downtime in any given month if they are to pay us.

So I have a plan, but I want to put it out in the open to see if anyone else see any problems with it that I may be overlooking, or maybe if you have a better plan you can suggest.

Our setup: Intel Modular Server, FreeBSD with jails, apache, mysql, php. The domain where the students log in is something like portal.mydomain.com

I was thinking that we could put up a second instance of our server at Site B (the new site) and give it a new domain name (portal1.mydomain.com). We could then force all traffic from portal.mydomain.com to redirect to portal1.mydomain.com. At that point we would make the DNS change for our original domain name to have the new IP address. Then we would give it 48 hours for DNS changes to propogate. At that point we would then just change apache on the new instance to respond to portal.mydomain.com instead of portal1 and then everything is business as usual.

Are there any holes in this plan I'm overlooking? Is there a better way to do it?

Safado
  • 4,726
  • 7
  • 35
  • 53

1 Answers1

2

For this to work you must be absolutely sure that your application does not depend on the current domain name and will work under portal1.mydomain.com.
My experience in the past is that sadly, this is not always the case.

If you can be sure that the domain name is not a problem, your plan sounds good.
You obviously need to synchronize the MySQL DB when you cutover to the new datacenter (you didn't mention that in your plan).
I would set the TTL of the portal.mydomain.com DNS record to a lower value, so it doesn't take 48h to propagate.

If you can't be sure, my suggestion of doing this would be the following:
* lower the TTL of portal.mydomain.com (5 minutes)
* setup a new instance in the new DC, also listening to portal.mydomain.com
* keep the MySQL DB in sync from old to new DC (replication)
* stop the Apache in the old DC
** wait for last DB changes to replicate
** stop replication on new DB, configure to be standalone
** change DNS record of portal.mydomain.com to point to the IP of the new instance
** configure the old Apache instance to access the MySQL DB in the new DC
* wait, monitor requests on the old Apache, shut down when not used anymore
* set TTL to a normal value again

This also assumes a few things like MySQL connections between DCs are possible.

faker
  • 17,326
  • 2
  • 60
  • 69
  • It does depend on the domain name but its an application we wrote. We put a global config file in it that allows us to set what the domain is so it shouldn't be an issue. I actually started thinking along the lines of your second idea as well and do mysql replication. Would it work to set up the db slave in the new site, replicate the data (is it instantaneous replication?) and set up the site exactly the same on each side. Then as DNS is propagating it won't matter which site they're going to as long as the data is going to both databases? We're currently sans DBA so I have some reading to do – Safado May 07 '12 at 15:59
  • Sorry, I guess how my solution differs from yours is to keep the data replicating while the DNS is being changed. In your steps you say stop the replication, then change the DNS. I may have students in Australia that perhaps would be going to the old site. so I'd need to make sure that the new site has all the data the old site had. – Safado May 07 '12 at 16:04
  • MySQL replication is asynchronous. I wrote to stop Apache (which effectively stops writes on the DB) and wait for the recent changes to replicate to your new DB. What I described is a master-slave replication, in which case you do not want to write to the slave while replicating. – faker May 07 '12 at 16:25
  • You're right, I read it wrong the first time. The database replication idea seems to be the better of the two options. We're also considering setting up an MPLS load balancer (it is our long term strategy to LB between two DCs) with only one site. So we'll change the DNS to the go to the LB, so as it's propagating, all users well end up at the original destination. Then once DNS has propogated and we've got site B set up, we just flip the switch on the LB and send all traffic to B, then shut off site A permanently. It all depends on if we can get a budget advance though – Safado May 07 '12 at 16:46
  • In which case, your solution to replicate the DB seems to be our best option (in the event that the LB idea falls through), so I'll mark this as answered. Thanks for the suggestion. – Safado May 07 '12 at 16:47
  • If you want to keep that LB, thats great. Similar you could setup a reverse proxy in front of it for the cutover, with the drawback that you need to switch the DNS record twice (to reverse proxy, cutover, to new instance). You're welcome! – faker May 07 '12 at 17:13