0

Tell me Guys, how would you transfer more than one million files, witch have a summarized size of about 150 GB from A servers (KVM) VM to another (let's call it B, XenCenter) servers other VM in maximum 2 days?

Currently I'm trying to compress them to an external Hard drive and then copy them on the B servers VM. However 2 issues have ocured:

  • Only the compression to tar.gz takes about 3 days
  • I'm unable to umount any hard drive from the VM, donno why. (CentOS 6.3) I can mount it up, I can see the files, everything. But after I give the umount command it hangs (waited 1h30m for a 16GB pendrive with one 500MB file to umount....)

The two VMs are unable to see each other and that cannot be changed. I'm currently open for any solutions that comes to your mind.

Bert
  • 984
  • 1
  • 11
  • 29
  • Can you not move the backing store for the virtual disk instead? – Spooler Jun 05 '18 at 14:48
  • Sorry, I can't understand what you mean. – Bert Jun 05 '18 at 14:51
  • If both of these are VMs, they store their data in some form of lower level storage in order to provide virtual disks. Those may be files, logical volumes, or LUNs from a SAN, etc. Depending on how the virtual disks are backed, you may be able to very easily copy those instead. Doing a file based copy of a million files with compression is going to crawl. – Spooler Jun 05 '18 at 14:55
  • 1
    `dd` or `rsync` (ext4,xfs) – alexus Jun 05 '18 at 14:56
  • You may try transfer you files over UDP, than use rsync to check consistency. It's just another solution. It depends on disc i/o and network. In theory, with 1 Gbps network 150G could take an hour. – Alexander Makarenko Jun 05 '18 at 15:23
  • 1
    How about just tar'ing them? Leave off the compression step and just allow tar to bundle all your files up into a single tarball (uncompressed) that can more easily be moved around. Alternatively, unmount then `dd` the block device to an image on removable storage, then mount on the other side? – dannosaur Jun 05 '18 at 17:23
  • I was thinking about tar-ing them just to move. However, neither dd, nor rsync is available since the servers will most likely be unable to see eachother. They both sit behind a line of firewalls restricting access in a number of ways. Unmounting the device from the KVM virtual machine is the issue. if that would be OK, i had no issues. – Bert Jun 06 '18 at 07:26

1 Answers1

0

Oh how simple it is.... a friend of me told me the answer.

Since they are both acting as webservers, I just tar the files without compression, move to the root folder of the website and I can simply download it to the B server with wget..... Yes... sometimes the outsiders have the best option. :)

Bert
  • 984
  • 1
  • 11
  • 29