I'm working on a website where we need to have a good deal of uptime. Specially in the short bursts (15-day periods) that the website's events happen.
The page is dead simple and can be served almost completely off an html cache. While there is a php based part, this is not mission critical and in case of failure, we can live with the cache for a say 20 minutes until the eventual problem is sorted out. More than 20 minutes won't really work as the site hosts a live score board among other things.
We had successful deployments in Amazon using multiple EC2 with elastic load balancing and also using the Rackspace cloud (Cloud Sites and Cloud Servers).
For this particular website we'd like to have the app running in these two providers at the same time but just serve the pages from off one of them. In case the main provider fails just move to the other and keep going.
I know how to make everything work in just one provider. What I'm not clear on how to achieve is the actual switch from one provider to the other. For example, if I CNAME myapp.com to a domain in Rackspace and it fails, when I change the CNAME to point to Amazon, a great deal of users will aready have their DNS cached to Rackspace and the whole thing would be pointless... this is one of the many questions that I have...
Any help is greatly appreciated... tips, advise, gotchas, anything is welcome...