11

At my new job, I have a production server with the following qualities:

  • Windows Server 2003, ancient hardware
  • Absolutely vital database
  • No backups whatsoever
  • Everyone in the company has full admin rights, the passwords are stored in a .txt on the global share
  • No installers, except for the OS
  • The machine itself is sitting on a wooden shelf 5 feet above the ground against an external wall with frequent truck traffic on the other side; the shelf is already bent from the constant load
  • Hasn't been rebooted in $DEITY knows how long, my predecessor wasn't even sure if it would survive it; of course no updates have been applied, ever
  • UPS is installed, but since everything is hooked up to it, it would last 10 minutes tops
  • No spare parts or hardware budget

How do I make a full backup with minimal impact on the server? I'm not sure how close it is to a total meltdown. For all I know, plugging in a USB stick could kill the company, and of course it will be all my fault, since "it was running fine before you touched it".

The ideal solution would be a VM, so I have a test environment as well (separate of course).

Update: After a long talk with the boss, I now have permission to do whatever I think is best in the long term, so there's hope now. Also, I found most of the backup-worthy data is stashed on the client machines in well-hidden directories, the server hard drive isn't big enough, and while I still don't have a hardware budget, there is another ancient box that has fallen out of use because all the malware bogged it down. Sigh.

HopelessN00b
  • 53,385
  • 32
  • 133
  • 208
György Andrasek
  • 615
  • 5
  • 9

4 Answers4

11

Everyone else's technical answers are perfect, but you need to consider another option; leave.

The reason I suggest this is because if your company values its IT that little then they probably value your efforts even less. When, and I do mean when, this system kills their business they'll totally blame you, firing you and badmouthing you where they can, maybe even coming after you for some form of damages. I could suggest you sitting down with them to explain their lack in investment and urgent need to do so but from what you've said I'm not sure they'll listen.

So my suggestion is to look for somewhere that deserves the professionism you're clearly interested in delivering - I don't think you'll find it hard. Good luck.

Chopper3
  • 100,240
  • 9
  • 106
  • 238
  • In complete agreeance here -- nothing good can come of this. – pboin May 10 '10 at 11:09
  • 5
    I agree as well, but this is still a much better job than the 12 hour shifts at the local cell phone factory. I live in a bad neighborhood. On the bright side, if I manage to clean this place up, I can handle anything, anywhere. – György Andrasek May 10 '10 at 15:04
  • 2
    Great attitude, best of luck. – Chopper3 May 10 '10 at 15:34
  • Accepting this answer for being correct on all counts. They basically wanted me to work a 8 hour shift at another position far away from any computers, tend to the system in my "spare time" (i.e. overtime for free), and take the blame for the inevitable meltdown. They also didn't have the guts to tell me about this part for two whole weeks. – György Andrasek May 28 '10 at 12:21
  • Oh dude, I'm so sorry to hear that, if it's any consolation I think you've handled yourself very well indeed from how it appears :) Keep at it and you'll go far. – Chopper3 May 28 '10 at 12:56
7

Nice.

I would recommend VMware Converter ( http://www.vmware.com/products/converter/ ). It's free, it will do physical to virtual (p2v), and I've used it to do something similar.

The only problem may be that it may be sensitive to the patch version of the target. I wish you much luck.

I can only recommend using this as an incentive never to allow a server to get into this situation. And I sincerely hope that your predecessor isn't in a position to allow something like this to happen again.

EDIT: It seems as though you know this is a bad situation, which is great. If you want to continue to hone your skills, I'd recommend picking up a copy of The Practice of System and Network Administration. It's Serverfault Approved

Community
  • 1
Matt Simmons
  • 20,218
  • 10
  • 67
  • 114
  • This seems like it's exactly what I'm looking for, thank you. Please don't blame my predecessor, he inherited the system as well, knows how bad this is, and offered to help with anything, if needed. He also did a really nice job with the cabling, and is apparently smarter than I am, since he quit. – György Andrasek May 09 '10 at 17:16
  • 1
    Echo the P2V idea - I've used it on similar ancient boxes as a way to grab an easy image and then bin the old hardware in favour of running the VM on some newer kit that we've got spares for. I'd be tempted to run a full NT backup on the XP machine first just in case it does object to VM converter install. – Chris W May 09 '10 at 18:23
2

Sounds like you're pretty screwed; the entire situation is absolutely horrific from an IT standpoint.

I'd probably do my darndest to get another server, get it up to date, install the database software on it, and secure it with proper permissions and get a good backup routine in place for the new server then migrate the old database to the new system.

To answer your question, if you're going to virtualize it there is a VMWare Converter that can do it, but you still have to risk issues doing this. You also have to be aware of some change-of-hardware items that occur in the conversion and getting it tested, as well as making allowances for the database and vital server being "down" during the transition.

This also doesn't fix the security issues (how do you keep some other employee from "Accidentally" deleting the vital database?) nor does it fix the lack of proper backup routine issue...

To answer your question more exactly, you can update it with any of a number of commercial backup programs that will usually involve running some kind of agent program that knows how to handle open files. It'll still involve reboots, though, so there's some risk from the description of the system you gave. You didn't say what this database was running so I don't know if there is a specific concern involved with the software configuration.

I'd be tempted to start off by scheduling a time to turn off the server and image the whole drive from a bootable CD, whether using something open source (linux boot disk and DD to another media or use partimage, but there's no guarantee these will restore properly to another drive without testing it) or commercial (acronis or ghost, for example). Once you have a image of the drive you at least have a chance of being able to restore or pull data off should it be necessary.

Whatever this business is, if they're telling you that the data and system is absolutely vital and necessary but they allow every user in the business to get full access to the data and possibly delete or damage data (intentional or not) and no provisions for properly protecting the data or supporting you when you say it needs to be done, then it's not absolutely necessary or vital. If they honestly believe that it is and they treat it like that, you may really want to consider this a temporary position, because you're going to be in for a rough ride.

Bart Silverstrim
  • 31,092
  • 9
  • 65
  • 87
1

Yes, use the VMware VM converter to convert it into an "infrastructure VM", prep a new harddisk on your server box with VMWare ESXi (currently at version 4.0 and it's free) as hyperisor, get it RAIDed (hardware) with level 6 or 10 if you're allowed financially as ESXi dun do SW RAID. Check if your RAID controller is one of those that works with ESXi here.

Then you can use VSphere to monitor it remotely.

Mind you that I got hit by the activation resurrection issue after converting a WS2K3(oem std) machine as it detected the HW was changed significantly, as long as your windows is not oem version I think you'll be fine.

Given no budget is allow, you may have to give up your own harddisk for the new VM setup and in return you can have the one on your server back.

But you should also brace yourself with the consequences, i.e. all fingers may start pointing to your corner from the moment you set up the new server because you just get yourself into it. I've been there done that just a week ago and I'm still in search for a good backup solution for both the VMs and database.

Good luck :-)

EDIT: For backup... vSphere has a datastore browser that allows you to do file copy of the VM files to your local machine given you are running the vSphere from your local machine. Make a snapshot of the running VM (use quiescence mode if situation allow) that free the locks of the VM files before copying, or else, you can use web browser to gain access to your ESXi-ed server and ftp from there within.

One thing to take into account is that the duration for pulling all VM files across network may slow down your server quite a bit.

user41653
  • 73
  • 1
  • 9
  • Does it work to take snapshots and copy those over as backups? I thought I had read that that was a bad idea to do it, that copying to another VM server and trying to run those images didn't work properly. I've been shutting down the VM completely and copying the virtualized system over the network to another system to make "machine copies", and if backups of the machine are to be regularly kept you use regular backup software that can restore from (virtualized) bare metal. – Bart Silverstrim May 09 '10 at 18:13
  • Yes, copy all files except the snapshots. Cut the long story short if you're restoring infrastructure type of VMs on to the same version of ESXi server then you'll have no problem. Here's the guide: http://www.vmware.com/pdf/vi3_301_201_vm_backup.pdf – user41653 May 10 '10 at 02:53