I'm trying to architect a set of web and database servers so that we can load test them. I'm running into some problems given the constraints I have to work with:
- We can't load test against production. Duh, mostly. We've looked at a responsible way of doing this, but the bottom line is that it just puts too much risk for downtime for our customers. So this means we have to have an alternate set of hardware to test against.
- To protect against what we're trying to protect against, we really need to have truly identical hardware, down to the driver versions running on each box. Merely approximating the bandwidth and software isn't going to do the trick. This means...
- The hardware is going to be expensive, so expensive that we can't afford to just have it lying around when we're not load testing it.
- This leads to having a temporary setup that we can provision on-demand. The downside of that is that it's still probably pretty expensive, even if it only runs for a week per month, and it has the drawback of being cumbersome and time-consuming. I'm trying to get load testing as a truly first-class artifact in our processes, and this would be a barrier.
If I could wave my magic sysadmin wand, what I would have would be a truly cloned, identical and load balanced architecture, with everything separated from the web application firewall on down to the database server. Thus, when I needed to load test, I could flip a switch that would move one half of the architecture from being load-balanced production into being a load test environment. Even this is not perfect, because it's only half of the capacity, and we're not testing whatever cluster management is going on, but Criminy...you see how crazy this gets.
I'm headed towards the conclusion "load testing on a regular basis just means doubling your hosting costs, period, or else accepting that you're not going to have an accurate hardware test". Is there something I'm missing here?