We currently have a simple 3 server setup:
- Cloud NGinx/Apache web server (NGinx serves static, dynamic is proxied to Apache/PHP backend)
- Dedicated Database server
- Cloud Outgoing email + monitoring server
We're at the point where we need to scale up/out our web server. We're looking to acheive the following:
- Minimise downtime if a web server fails
- Offer additional capacity during peak times
- Allow us to upgrade one web node without taking down website
- Allow SSL serving also
I would like to add an additional web node, with equal capacity to the original one, rather than doubling the capacity of the existing web node, as this will give us a failover/mirror node (when load balanced) to improve capacity and offer redundancy.
I've looked into the options in depth, at the moment HAProxy seems to be out of the question as I want to handle SSL (and don't want to terminate at the Load Balancer, all SSL traffic must be encrypted end-to-end).
My options seem to be:
A: Add a software load balancer (eg Nginx) on it's own node, in front of the two web nodes. Issue here is when UGC is uploaded by a user, it won't be available on the other node. With sticky sessions this is less of an issue but ultimately we need to sync uploaded content otherwise the nodes will become out of sync. Can use Rsync but as we're looking at master-master web config, need bidirectional..tricky!
B: As above, but use the load balancer to also serve static content. This would then offload some of the processing from the web nodes. We could rsync all uploads from the web nodes to the load balancing node, this way it's only a one-way rsync from web->load balancer. Any non-static requests are load balanced to the web nodes. This keeps all static content in one place but I was aiming to have a small load balancing node and this may end up growing quite big?
c: As A, but use NGinx to direct any visitor traffic where uploading is likely (this is easy to segment) to one 'master' server to do their uploads, and fairly load balance between the other nodes for 'read only' traffic. This makes syncing easy as it will be one-directional (master web node->slave web node), allows us to use the disk space on the web nodes better and keeps static content off the load balancer. However it means that if the master node goes down, we'd need to manually switch traffic for our 'upload' areas to the remaining server (nginx may be able to do this switch, we just have a different load balance config for our upload areas)
B seems like a good config, and I could probably combine the Load balancer node with our existing email/monitoring server as it sits idle. There is a single point of failure with the load balancing node, but in an emergency we could bring up a new cloud instance to replace this. However I don't like the idea of the static images only being on one node as we'd have no inherent backup. I also like the idea of C as it means it's kept very simple. If the load balancer fails, we could direct traffic direct to one of the web nodes without any issues (as they would both have a full set of static files).
Any input on the solutions above would be very much appreciated