2

We have a media-rich web application that is hosted on AWS. We have several Web Servers and we have an NFS server.

On the NFS server (Linux server) we have several EBS volumes that are mounted and we've used mdadm to implement the different mounted volumes as a single RAID volume. The Web Servers simply access the NFS storage through a mount point.

Amazon has now let us know that they will be performing power maintenance on this server in a couple of days time. Since all our media is on here it would render our site unusable for the hours while Amazon is working on it. We want to try and prevent this downtime.

I was thinking that we can prevent server downtime by perhaps setting up a new server temporarily and attaching the EBS drives (raid volume) to that server and have our web servers point there during maintenance.

This is a very high risk operation since this involves several terabytes of our production data.

What would be the safe way to move over our logical raid drive (md0) to a new amazon instance? I was hoping that I could start with building the new server, mounting the ebs volumes and assembling the RAID partition using mdadm --assemble --scan before unmounting from the existing instance so that I can first test that everything works and thus having it mounted on two instances at the same time, but I don't believe that is possible with the way that filesystems work.

How do I move a Linux software RAID to a new machine? suggests a way to move drives, but isn't really a cloud-based question. Perhaps there are simpler ways to prevent system downtime with our solution being hosted on the cloud? I have considered taking an EBS snapshot, but that tries to replicate all the many terabytes of mounted storage, so this is not a practical solution.

Any ideas?

Community
  • 1
Stanley
  • 365
  • 2
  • 4
  • 14

2 Answers2

2

I just realized: having to manually create a new instance and manually move over the RAID drive is unnecessary.

Since the instance is EBS based and the drives that make out the raid volume is also EBS based I can simply stop the ebs image and then start it again. This will migrate it to new hardware.

This is by far the simplest way to do it. Since the server will now receive a new IP address, I just have to update fstab on the web servers to point to the correct server.

Stanley
  • 365
  • 2
  • 4
  • 14
  • Did this work? I'm in the same situation that you were in (except that I use RAID10). Just stopping the EBS instance and starting it again after a few minutes moved them to new hardware? – Phalgun Apr 24 '15 at 12:36
0

You can only attach an EBS device to a single instance, so you're going to have to detach it when moving. I'm assuming that you want to avoid data intensive processes like creating an EBS snapshot or rsyncing the data to a new instance. I'm also assuming that you're using RAID1.

The safest option will require a couple minutes of downtime. You would start up a new instance, and install and configure the software necessary (e.g. NFS server). Then on the old instance, unmount the filesystem, stop the array and detach the two EBS devices. Then attach the EBS devices to the new instance, start the array and mount the filesystem. Get the webservers to mount NFS from the new instance. Starting the array should just be a case of running the mdadm command you describe, but I'd definitely test this first.

The second option potentially has lower downtime (assuming that you can operate in read only mode for a while), but is more dangerous. You'd start the new instance as above. On the old instance, remount the filesystem in read only mode. Then fail one of the RAID devices, detach this EBS device and attach it to the new instance. Start the array in degraded mode on the new instance, mount the filesystem, and get the webservers to mount NFS from the new instance (the site should be fully available at this stage). Then stop the array on the old instance, detach the EBS device and attach it to the new instance, and add it to the array. This may however trigger a full resync, so again, test this first.

Whatever you do, make sure to test the process first so you know exactly how to perform it, and make sure you have backups in case it goes horribly wrong. (Also, consider storing your media on S3.)

mgorven
  • 30,036
  • 7
  • 76
  • 121
  • 1
    Thank you. I think I'd rather go for option one. Just to clarify, when we first created the raid device we ran mdadm --create --verbose /dev/md0 --level=0 --raid-devices=3 /dev/sdf /dev/sdg /dev/sdh (which I know **NOT TO DO** with new instance since it will erase my data) and then had to do mkfs -t ext3 /dev/md0 to create an actual filesystem. We then mounted /dev/md0 to a mount point. If I create a new server and want to attach existing raid drive, what alternative do I use for the mkfs command? (For mdadm I know to use mdadm --assemble --scan) – Stanley Jun 12 '12 at 08:13
  • @Stanley You're using RAID0, so option two isn't an option anyway. You don't run any `mkfs` command -- just mount the filesystem. – mgorven Jun 12 '12 at 08:31