12

Okay so it's a challenge and I'm having trouble planning it mostly due to lack of experience. However it's become a task in my job and I'm doing it mostly solo with an extra hand.

I'm comfortable with Linux and all these servers are Debian or CentOS. Some are VPS, Shared hosts and other are Cloud VMs. I'm new to the company and have only gotten time over the last several monnths to take stock of outstanding issues. They all run plesk or cpanel.

I've found these servers were never configured, just bought and sites thrown up on them with very basic administration in the last two years (disk space clean up and domain/hosting management tasks mostly). There was no iptables set up and no security precautions taken and so these servers are riddled. I figure the best solution here is to start fresh and consolidate our hosting to fewer servers as they had just purchased new shared servers willy nilly. I feel we can support our entire client base on around 1/4 of our cluster.

My problem is that I'm not sure how to plan this move. It will require a LOT of moving sites and DNS changes. I'm thinking it would be wise to seek advice from any wizards out there who deal with this sort of thing regularly or at least have the experience to point me in the right direction.

Mud

HopelessN00b
  • 53,385
  • 32
  • 133
  • 208
Mud
  • 133
  • 7
  • 7
    " I'm having trouble planning it mostly due to lack of experience." You *absolutely* need to get yourself a copy of The practice of system and network administration (aka the Limoncelli book). Its probably got step by step guides to nearly every task you're likely to face as a sysadmin – Journeyman Geek Dec 10 '13 at 01:08
  • 2
    And I wouldn't badger the point but if you can avoid it, cpanel is pretty notoriously insecure. If you can, manage things the 'proper' way. Of course, it all depends on your needs. – Journeyman Geek Dec 10 '13 at 01:31
  • Thanks very much for your help, your solution below has given me a good framework to work with. I'm inexperienced but I'm not a noob. I think I can tackle this and time is on my side thankfully. I will be picking up a copy of that book - it sounds like the perfect new addition to my desk library. (It grows so fast :| I never thought!) – Mud Dec 10 '13 at 01:57
  • I *just* finished up the last module of a cyberforensics and information security management major. Its the book I *wish* I knew about earlier. That, Mandia and Prosise's Incident Response and Cyber Forensics and Eoughan Casey's Cybercrime and Digital Evidence are what I recommended to my classmates as essential reason. Its an *awesome* book, thats likely to follow me to my next job, and maybe the one after that. – Journeyman Geek Dec 13 '13 at 00:13
  • [Administration panels are off topic](http://serverfault.com/help/on-topic). [Even the presence of an administration panel on a system,](http://meta.serverfault.com/q/6538/118258) because they [take over the systems in strange and non-standard ways, making it difficult or even impossible for actual system administrators to manage the servers normally](http://meta.serverfault.com/a/3924/118258), and tend to indicate low-quality questions from *users* with insufficient knowledge for this site. – HopelessN00b Feb 25 '15 at 08:46

1 Answers1

19

There's no simple solution to this - its probably going to be a long and annoying road to getting everything moved. I'd start with a few things, but the book I recommended in the comments likely goes into a few details. I see four main stages here - inventory, redeployment, testing, and acceptance.

  • What am I hosting right now

If you haven't already, do a complete inventory of what you have and what you're using now. Identify what are critical resources, and what are just sitting there. This question on SF may be a good start, though naturally, you will need to adjust it for a debianisque environment - maybe like this. This is the most critical stage.

'Web' servers are not all the same - check if you're running some process that runs its own web-server (like django) or if its a standalone webserver like apache.

  • re-evaluate the current setup for best practices

Are you using EOLed versions of your OS and other software? Can you migrate? What issues would you face? Are there any known security risks? Can I standardise everything to a single version of all my software? Can I automate my deployment processes and to what extent (look at puppet and chef).

  • create a set of tests for each server

Make a list of what needs to work, and how you're going to test it

  • decide what you're moving to

That way you can pick the best P2V or V2V process for your needs.

  • decide what to consolidate

You may choose to run several set of services that were seperate before on one server. You may also choose to run VMs. Both running very task specific VMs (you can restart one without affecting other things), and grouping together similar tasks (so you have fewer logical systems to maintain) have advantages

  • set up an isolated testing environment

This way you can set up and test a system move without it getting in the way of production. You're likely going to be able to do this on maybe a single server running VMs.

  • have a staging area for tested systems

Move the VMs over to the staging area and test them one last time

  • Make use of maintainance windows for cut-over to the new systems.

Set a short dns ttl for systems that have been migrated successfully and change the domain name and other relevant settings during a maintenance window. Test again. Keep an eye out for any errors once the new server has been deployed

  • Do not destroy old server installs until you can verify the new one works

Cover yourself. If something does go wrong, in the short term, you can always fall back to the old server.

Community
  • 1
Journeyman Geek
  • 6,969
  • 3
  • 31
  • 49
  • Thanks very much Journeyman Geek although really I'd consider you an Artisan :D – Mud Dec 10 '13 at 09:48
  • 3
    +many, if I could, for all of this. The only thing I'd add is that, as this will take a while, get management buy-in to a period of stability while you plan, test, and execute the move. During this stability period no new web servers may be created, and no change that you do not personally approve of and implement may be made to any existing server. Otherwise, you're trying to carefully and professionally copy a crazily and stupidly moving target. Think also about how you'll control change on the servers after you've done the move, or you'll have to do this all again soon enough. – MadHatter Dec 10 '13 at 09:50
  • Nice point MadHatter, I have a good line of communication with the the higher ups, they understand that security comes first with regards to our clients data and that this process is unavoidable and that it can only be bandaided for so long. – Mud Dec 10 '13 at 23:55