dd file copy uploading to server is fast, downloading image back to local machine is slow

I've used dd to copy a hard drive on one server to the samba share on another server. The speed was between 30-60 MB/s, about what I'd expect. The connection is a dedicated gigabit line for each image.

However, when I try to read the image from the server, and write that back to a drive here on the machine, it reads at a high speed (~90-100 MB/s) then quickly drops to 0 for a second, gets about 16MB worth of data, drops to 0, gets about 16MB, etc, etc. It's really slow overall, and I can't find out what the issue is.

I've tried using both samba and NFS on the backup server, tried playing with async, rsize, wsize, and they might change the initial speed, or even the speed when it does move, but it still has large windows that it transfers nothing.

In addition, I've even tried

dd if=myShare/myImage.dd of=/dev/null

and I still get the fast speed for 2 seconds, then drops to near zero. I believe it is a limitation with the backup server not being able to upload the data as fast as the client is requesting.

What can I look at?

Here is a tcpdump of me trying to copy the file for 60 seconds.

UPDATE

I just did a read test on the server.

dd bs=4M if=sda.dd of=/dev/null

that command was fast initially, but gets to about 1.7 or 1.8 GB (As reported by dd) and hangs up and stops copying.

The volume that that file is on is a RAID10 on an LSI controller.

McKittrick Swindle

Posted 2016-07-14T19:29:26.660

Reputation: 11

Can you upload a tcpdump/Wireshark capture demonstrating the issue to Dropbox or something? – Mark Riddell – 2016-07-14T19:38:06.680

here's a link. I've never tried to read a tcp dump before.

The server is 172.18.222.1, and the client is 172.18.222.101. ...101 is requesting the data from ...1

– McKittrick Swindle – 2016-07-14T19:51:55.503

Thanks. Can you run the capture again but save to pcap rather than txt so that I can open in Wireshark. Please also try and run the trace from the very beginning of the transfer so that I can see the TCP handshake. – Mark Riddell – 2016-07-14T20:26:47.333

Thanks for the help. I think I've narrowed it down to being to storage volume. It is a RAID 10 with 4 2TB drives in them. hdparm says its only got ~900kB/s, and dd gets very slow after a few seconds. This is from reading a file to /dev/null – McKittrick Swindle – 2016-07-14T20:28:48.410

Cool, sounds like you've found your smoking gun. Good luck! – Mark Riddell – 2016-07-14T20:33:35.783

dd file copy uploading to server is fast, downloading image back to local machine is slow

No answers