Migrate raw disk image over LAN

Question

Here is my situation:

• Two dedicated servers in the same datacenter with gigabit ethernet between them.
• Both dedicated servers booted into a rescue environment based on Debian Squeeze with extra tools and utilities added. Also plenty of tmp space (32GB of RAM on both boxes) for downloading software, installing packages, and/or compiling as needed.
• Both dedicated servers have approximately 3TB of usable space.
• The "source" server has 4 x 1.5TB disks in Hardware RAID-10 with an Adaptec 4 port controller.
• The "destination" server has 2 x 3TB disks in Hardware RAID-1 with an Adaptec 2 port controller -- same generation as the other, but different number of ports.
• The number of usable blocks on /dev/sda differs by less than 10 MB, but the destination server's array is for some reason a few megs smaller.
• Both RAID arrays are configured to use the entire disk surface of all constituent disks to create one, single RAID volume.
• The operating system boots in MBR mode; no UEFI booting is used.

What I want to do:

• Copy, at the block layer, the entire OS image (this only consists of GRUB2 bootloader in the GPT partition table, /boot partition, and / partition) from the "source" server to the "destination" server.
• If possible, the copy should take place "live": this means I don't have enough space to store a proper file of the disk image on the destination side, unless I'm unpacking the disk image onto the hard disk as the copy is taking place. The gigabit ethernet connection between the servers is reliable enough that I'm comfortable with this, and I will of course run fsck on both ends (source and destination) to verify the filesystem is OK before and after the transfer.
• If possible, do not transfer blocks over the network, which are not used by the constituent filesystems in each partition (all partitions are formatted as ext4). This is because more than 50% of the "source" disk is free space in the / partition.
• Adjust the size of the / partition so that when it is copied, it is resized to fit within the just barely smaller size of the destination disk.
• Once the copy is successful, mount each volume and fix up references to static IPs to reflect the IPs of the new server. (Can do this just fine without any further help)

My questions:

• Should I first calculate the difference (in bytes) between the size of /dev/sda on each server, and then use e2resize to non-destructively reduce the size of the / partition on the source side so that it will fit into the space of the destination side?
• Should I run dd on the raw block device, /dev/sda from the source to the destination (over ssh), or should I create an equivalent partition layout on the destination and run dd on each partition? Note that handling a partition at a time leaves me the problem of the bootloader, but if I don't do it a partition at a time, then dd needs to know to stop transferring data once it has written as many bytes as the destination can hold (which hopefully will "close out" the very end of the / partition on the last block, which is logically "to the right of" all other partitions in the partition layout of the source).

A few misc. specifics:

• The host OS on the source box is Ubuntu Server 12.04 running several OpenVZ guests
• Since both boxes are booted into rescue, direct disk access is possible without expecting any change to the underlying data by the running operating system.

Answer 1

This is messy, but doable.

I presume here that / is on /dev/sda3 and that /boot is on /dev/sda1.

1. Shrink the filesystem on the old server to its minimum possible size.

   oldserver # resize2fs -M /dev/sda3
   

2. Partition the new server's disk with an identically sized /boot, swapspace, and new / partition (and anything else you need).

   newserver # parted /dev/sda
   

3. Copy the / and /boot filesystems.

   oldserver # dd if=/dev/sda1 | ssh root@newserver "dd of=/dev/sda1"
   oldserver # dd if=/dev/sda3 | ssh root@newserver "dd of=/dev/sda3"
   

   Because the partition on the new server will be slightly smaller than the one on the old server, you'll receive a spurious No space left on device message at the end of this. However, since you shrank the filesystem at step 1, this doesn't matter.

4. Resize the filesystem on the new server to the size of the partition.

   newserver # resize2fs /dev/sda3
   

5. Install GRUB on the new disk.

   newserver # mount /dev/sda3 /mnt
   newserver # mount /dev/sda1 /mnt/boot
   newserver # mount -o bind /dev /mnt/dev
   newserver # mount -o proc proc /mnt/proc
   newserver # chroot /mnt /bin/bash
   
   newserver(chroot) # grub-install /dev/sda
   newserver(chroot) # exit
   

6. Finish the rest of your fixups (IP address, etc.).

You can probably find a way to avoid copying the partition's free space, but it'll probably take you longer to research than to just copy it all...

Answer 2

I'd mkfs fresh filesystems on the new server, then rsync them from the old server. That is restartable, consistent, and each file is easily individually verifiable. Where you're discarding unused sections of the filesystem (not a forensic copy), I don't see any reason to not use this method. You would have to re-run GRUB, but that shouldn't be a challenge.

Explaining a raw copy that is file-system aware would take me a while, so unless you comment as to why my rsync solution doesn't work I'll spare myself the typing.

Answer 3

If you REALLY want to transfer data at a block device level, I can think of one pretty useful trick I was using to migrate servers with minium downtime involved.

The thing is, you can create a degraded mirror on source server with your data partition being the only active half of the mirror, then export destination partition from second server via AOE (I suppose both your servers are in the same broadcast domain). At source server you then connect network block device to your degraded mirror so it would start rebuild. Wait until rebuild is complete, stop your mirror, remove AOE exported device and you're OK.

A bit more details follow (I'll try to keep it brief).

Components:

• mdadm with its build mode (ad-hoc mirror without metadata);
• vblade for exporting block device as AOE network device;
• aoe-tools for importing AOE network block device.

You have to create partition table on your destination server, then shrink source partition so it would fit destination. You can easily install GRUB to your new MBR; syncing just partitions over newly created partition table is a bit less error-prone.

On the receiving side you have to export your partition with vblade tool, on source server you can see exported devices after installing aoe-tools (run aoe-discover then look at /dev/ether/ for devices).

Then you should build raid1 device on source server with your source drive:

mdadm --build /dev/md0 -n2 -l1 --force /dev/sda

After this you can examine newly built mirror:

mdadm --detail /dev/md0
cat /proc/mdstat

At this point you can safely attach exported destination partition to this mirror:

mdadm /dev/md0 --add /dev/ether/eX.Y

Then just watch over synchronization progress:

watch -n5 cat /proc/mdstat

After sync is done, just stop the mirror: mdadm --stop /dev/md0 on source server, terminate vblade process on destination server, install GRUB on second server, change your IP addresses, etc.

Actually, with this trick it is possible to move server between boxes almost live, with downtime just to reboot synced boxes.

---

For performance reasons, I also suggest you increase your link's MTU (or set up a separate VLAN with jumbo frames enabled, if possible).

Note, you also can use something like nbd-server/nbd-client (or even iSCSI, if you want it rough) as an alternative to AOE, but AOE (vblade + aoe-tools) have a very simple interface and a great performance (no TCP/IP overhead),