0

Please, excuse my horrible english. I'm triyng to find the way to clone a HDD server to a remote destination location, using SSH and, if possible, compress data on-the-fly to minimize unnecessary data transmission between them. The main objective is a server migration between physical & VPS appliances.

Broadband is approx 50 - 60 MB/s (transfers at 6-8 MB/s). Running a full-dd-copy estimate a 28 hours copy time. I want reduce that time as much as possible without reduce partition size in source server (i want source as-is).

Origin & destination servers are in offline status (both initiated in rescue mode).

Data on original server are 60 GB. Rest of partition is filled with zeros.

There is a way to archive that?

Until now i cloning entire disks (but not so large) with this command:

dd if=/dev/sda status=progress bs=10M conv=fsync | ssh user@ip dd of=/dev/sda

That works, but this time i'm trying to do it with too much data...

Any ideas?...

Thank you very much in advance to all. Best regards!

D.

Diego Ferrarese
  • 11
  • 3
  • aslong as you don't describe the network and other related things, it will only opinion based things possible – djdomi Jul 05 '22 at 16:55
  • Hi, djdomi... What things do you need to complete the question?... Source & destination servers are In different data centers from same enterprise. Have root access in both servers... – Diego Ferrarese Jul 05 '22 at 17:44
  • hm, why not use nc for that job, it will pipe the raw data without the hazzle of encryption, second option is use a temporary space to dump the file – djdomi Jul 05 '22 at 20:25
  • I was thinking about it, but SSH is better because data encryption. Precise, i don have so much space to temporary dump data, and time factor is something important in this scenario...because of that i thinking sending data with -C flag in ssh, compressing data on-the-fly... but also i'm unsure about conv=fsync in DD... or perhaps conv=sync?... my doubts are in copy of "assigned blank space" more efficiently in network... – Diego Ferrarese Jul 05 '22 at 21:06
  • What is the file system? I mean, using tools like `e2fsimage` might reduce the transfer to only used data blocks. But those tools are, of course, file system dependent. – Nikita Kipriyanov Jul 06 '22 at 03:19
  • So are you ultimately asking for compression? If so you can just use `gzip -c` and `zcat` in addition to / instead of `dd`. (Certainly you can use any other compression utilities that compresses even better and works with piping.) Just make sure you quote the whole comnand line for the remote host, i.e. `'zcat | dd of=/dev/sda bs=4k'` / `'zcat > /dev/sda'`. – Tom Yan Jul 06 '22 at 04:07
  • Does this answer your question? [What is fastest way to copy a sparse file? What method results in the smallest file?](https://serverfault.com/questions/665335/what-is-fastest-way-to-copy-a-sparse-file-what-method-results-in-the-smallest-f) – Gerald Schneider Jul 06 '22 at 06:39
  • Try `dd if=/dev/zero bs=1M count=100 conv=sparse 2>/dev/null | wc -c`. `conv=sparse` "suppresses" *writing*, not reading. (*What has been read* determines whether that should actually be written.) – Tom Yan Jul 06 '22 at 06:58
  • File system is ext4. Thanks guys... I'll try Tom Yan's suggestions and come back with an update shortly... Thank you very much & Regards! – Diego Ferrarese Jul 06 '22 at 17:35

2 Answers2

1

Ok, guys... finally, server migration was executed with status SUCCESS! Data copy time was aprox 5hs 30Min. Command & total time:

dd if=/dev/sda bs=5M conv=fsync status=progress | gzip -c -9 | ssh user@DestinationIP 'gzip -d | dd of=/dev/sda bs=5M'
751593062400 bytes (752 GB, 700GiB) copied, 19185.8 s, 39.2 MB/s

I want to thanks to all for your comments, specially Tom Yan. Hope this help someone else! Best regards!

Diego Ferrarese
  • 11
  • 3
0

Ok Guys. I return with new findings. Finally, I think i got it. Thanks to Tom Yan's directions, i realized that piping gzip -c at source side, and gzip -d at destination side did the trick.

I tried -C flag in ssh connection, with relatively no impact in speed.

So, after several tests this was the command that resolved my problem:

dd if=/dev/sda bs=4MiB conv=sync status=progress | gzip -c -9 | ssh user@DestinationIP 'gzip -d | dd of=/dev/sda bs=4MiB'

As Tom Yan's post, firstly (my bad) i not quote whole command line for the remote host and the the result not changed... so, i don't understand the reason of that quotes... of course, finally i put them... but not understand why...

Diego Ferrarese
  • 11
  • 3
  • If you do not quote the command line for the remote side, the local shell will interpret the `|` in it, which means you would be piping the output of `ssh user@DestinationIP gzip -d` to `dd of=/dev/sda bs=4MiB` run on the local host. (Luckily the source and destination are both `/dev/sda`, so hopefully that didn't cause you any unexpected data loss...) – Tom Yan Jul 07 '22 at 21:31
  • Instead of quoting, you can also use ```\``` to escape any `|` or `>` (that are after `ssh user@DestinationIP`). In that case the command line for the remote shell will remain being passed as multiple arguments (instead of one single argument) to `ssh`. (I'm not that familiar with `ssh` to tell whether it could make a difference in certain cases.) – Tom Yan Jul 07 '22 at 21:37
  • 1
    Oh! Excellent explanation! Thank you very much! Tomorrow night will executed real job (hope successfully). Thanks again & come back with final results! Regards! D. – Diego Ferrarese Jul 07 '22 at 21:58