0

I am looking for options and solution for my company project where we can host our websites on multiple servers and we can use load balancing. The major issue is that when I update websites, I need to restart server. Our web projects are written in Djanog/Python and it requires uWSGI restart everytime I push updates. Currently we are running our websites on single servers and it is hard to update/restart while clients are using the application. I have been looking for solution on the internet and I found a good article on DigitalOcean's website. You can read more about it here.

We are also looking to separate our database from running on same application server and use on specific machine and also master-slave setup for replication. So my question is that for instance, our site example.com is running on web servers/ip 101 and 102. These both sites are connected to database ip 201. So now, when I push update on server 101 and restart the application, I want 102 to handle all the requests without throwing any error or anything. After 101 is updated, I want it to kick back in and handle all requests while I update server 102. I will also have same setup for database so 201 and 202 as masters and one is updating, the other one kicks in.

Our current servers are running Ubuntu and Nginx with MySQL serving Django application. I may not be familiar with industrial terms so I am explaining the question/problem. I am not talking about companies like Google, Microsoft or Facebook as I know they have their custom solutions for this kinda problem but can anyone guide me to look for what by terms or any explanation, article, post or tutorial that I can read to handle this solution?

Thanks

Python Programmer
  • 101

2 Answers2

1

You have two main options for taking servers down

  • You can take the server out of the Nginx load balancer and reload the configuration
  • You can rely on health checks to stop sending traffic to servers that aren't available. Info here.

You may need to be a bit careful with databases.

  • Will it be master / slave? If so when the master is down you can't do updates
  • Will you replicate between the two databases? Will that cause any consistency problems? Do you need sticky sessions?
  • Will one database be dedicated to one web server? This makes updates easier but replication more difficult.
  • If you update the database schema on one database then bring it back up how does this work with replication?
  • Could you use blue/green environments? ie one web / database server is updated to the new application version, then you send an increasing percentage of the traffic to it. This only works with stateless web servers.

I would probably do Nginx with health checks, application servers aware of both databases with failover code, database replication, and blue/green deployments. Be sure to test failure scenarios well.

In AWS you'd simply use:

  • An elastic load balancer
  • Two stateless web servers
  • AWS Relational Database Service. This keeps a master and a synchronous backup. It doesn't however solve the problem of database schema updates. Blue/green would help here.

Updates

The database should be in same location as app/web servers to reduce latency. You could move your whole environment to a cloud service if you wanted to, but you need to understand the environment rather than charge in and hope for the best.

Regarding the database master/slave setup, I'm probably not the right person to advise. You'd have to consider multi master, master / slave, and an environment where there's no master. Then there's NoSQL to consider, which can be a good fit in some cases.

I would probably look at master/slave, with both app servers looking at one database and the other staying up to date. When you wanted to update the schema (which is hopefully rare) you could stop replication, update the slave, then point the new version of the app at the old slave, new master. In the cloud you'd just probably just build new environments for each deployment - servers as cattle, not pets.

Tim
  • 30,383
  • 6
  • 47
  • 77
  • Thanks for the help @Tim. I will look into more info for health checks. About the database, I was thinking to keep 201 and 202 masters and one more server, 203 as slave. What would you suggest for database? I mean should we go with our servers and host db on them or use AWS or other cloud services? I mean if db hosted on our servers, I think we can deploy future projects and their db easily! Or migrate and update current db. I have been handling single instance easily but now I need some kind of industry level solution so we can be more reliable for our clients. – Python Programmer May 22 '17 at 22:47
  • Updated answer. The main thing I can tell you is you're probably going to need to do some learning or hire someone (staff or consultant) who's experienced in this area. – Tim May 22 '17 at 23:00
  • Thanks for help. I will look into more stuff you suggested! – Python Programmer May 23 '17 at 15:01
  • It's not exactly a full answer, but it gives you something to think about. – Tim May 23 '17 at 18:48
  • Yes! I have been searching about it and I think I still need to research more. But thanks for the help. :D – Python Programmer May 23 '17 at 19:54
0

The method to update one server, while the other server is still serving, is called "Blue-Green Deployment". You can read information about it at Martin Fowler blog here: https://martinfowler.com/bliki/BlueGreenDeployment.html

Let's say you have web1 and web2 servers

The steps that you need to take is:

  1. Remove/disable web1 from the nginx load balancer, so your customer will only go to web2
  2. Update your web2 server with new code, and restart web2 webserver. Test the application in web2, making sure it is serving the latest code
  3. Add/enable web2 into the nginx load balancer. Now your customer have 50/50 chance of getting new website or old website.
  4. Repeat step 1-3 for web1 server
  5. Now your customer will get only new website

I'm not that familiar with database, but if you use master-slave style for database setup, you can configure your website in maintenance mode where it can only read data, but not write data from slave server. Once you have updated the master, change a variable in your app to point to master, and enable back read/write

Sharuzzaman Ahmat Raslan
  • 376
  • 1
  • 4
  • 15
  • Thanks for help. I will look into more stuff you suggested! – Python Programmer May 23 '17 at 15:01
  • 1
    The approach I use is similar to this, but I do it at the module/service level instead of at the top level. That allows for easily doing localized bug fixes when there's no API change. When you do have an API change, you can do the same thing but with more than one module. – Diogenes deLight May 23 '17 at 21:37
  • 1
    Where anything like this becomes problematic is if you have a client-server API change...for example an additional JSON param or a new (required) POST field. In that case you'll need to either allow a scheduled downtime (even big websites do this...it's not the end of the world), or add a version-updater module to your new version. That's an inherent need; you can't architect around that. – Diogenes deLight May 23 '17 at 21:38
  • Thanks @DiogenesdeLight for the comment. We are using API for our service but also, we have control over the app. No third party is using our API. But thanks for the help – Python Programmer May 24 '17 at 15:07
  • Sure. I actually meant something different though. By client, I mean your users' web browsers :-D If you add a form field and they submit the old form without that field (because they didn't refresh their page), what happens? Or if their browser's js engine receives unexpected json because the backend (your app) suddenly sends json with different field names -- that needs to be handled in the page that your user is viewing. One way to handle, for example, is always jump to a reload page. May not be obvious now but at some point this will matter. – Diogenes deLight May 24 '17 at 16:32