Transfer 300+ GB at maximum speed on two headless servers?

Question

I am currently trying to transfer over 300GB from my Mac Pro server to my Synology NAS. I only have my Windows PC I can work from, but I can SSH into both devices. They are both connected to gigabit ethernet, which I have tested and which works very good.

I have tried a few things already, such as SCP, FTP, SMB, and AFP, but they are generally limited to about 30 MB/s. 30 MB/s would mean, theoretically about 480 Mbps on my wired network. I should be able to do ~60 MB/s, as I need to both "upload" and "download" at the same time.

10GB takes about 5 minutes to transfer, meaning I probably need a good 2.5-3 hours before all my files are done transfering.

Is there a faster way than simply using this AFP, as I am doing right now? Maybe I hit a bottleneck somewhere, but I can't really see where that would be.

Where are you using those SCP / FTP / SMB / AFP? I would start by using SCP from the Mac Pro shell to the NAS. — Tero Kilkanen, Sep 06 '17 at 14:36
@TeroKilkanen Yes, I am connecting to the NAS from my Mac Pro. I SSH into my Mac Pro and run the commands there. — MortenMoulder, Sep 06 '17 at 14:38

score 2 · Answer 1 · answered Sep 06 '17 at 14:39

2

Are you sure the limit isn't from the HDDs?

SSH with a faster cipher can help:

ssh -c arcfour
scp -c arcfour

Or you could run rsync without sshd, with rsyncd.

Another option is to pipe tar over netcat:

On the receiving end do:
# netcat -l -p 7000 | tar x

And on the sending end do:
# tar cf - * | netcat otherhost 7000

(taken from http://toast.djw.org.uk/tarpipe.html )

answered Sep 06 '17 at 14:39

ptman

27,124
2
26
45

I'm getting an "no matching ciper found" when trying to use the `-c arcfour` with SCP. I'll try rsync right after I've installed ipkg, so I can install netcat haha. – MortenMoulder Sep 06 '17 at 14:51
man ssh: The list of available ciphers may also be obtained using "ssh -Q cipher". – ptman Sep 06 '17 at 14:52
And arcfour is listed there – MortenMoulder Sep 06 '17 at 15:07
What about the server you're connecting to? – ptman Sep 06 '17 at 15:07
2

Okay, doing `scp -o Cipher=arcfour` made it work. Now I'm doing about 10 MB/s, so that's definitely an improvement in SCP. No where near the speed of SMB, though. – MortenMoulder Sep 06 '17 at 15:08
Both servers show the same ciphers – MortenMoulder Sep 06 '17 at 15:08

Transfer 300+ GB at maximum speed on two headless servers?

1 Answers1