Fastest method to automatically transfer large encrypted files over 40Gbit LAN?

0

Building on a couple of threads like this or this I am looking for the fastest way to regularly copy files in an encrypted fashion between two Linux computers. This has some special limitations which is why I decided to open a new question on it:

  • The computers are connected via a 40Gbit (and later possibly even 80Gbit) LAN. Latency is not an issue as we are talking about a fiber, switched interconnect of 20 meters or so.
  • The assumption is that the sending and receiving storage end can saturate the 40Gbit connection
  • The transfer should be encrypted. Only if it would really bring down the speed, encryption might be disabled
  • The transfer needs to happen a couple of times per day. Hence, we need to be able to automate it in some way.
  • The files to be transfered are probably in the range of 20 - 250 GB each

From what I learned so far, netcat is probably the fastest way. But is it suitable for unsupervised, automated use? And what about its speed when tunneled over SSH? Is there any encrypted protocol which can saturate a 40G or eben 80G link?

//edit: As requested some details on the hardware:

  • The servers are HPE Proliant DL380 Gen9
  • The Storage Controller is a HPE Smart Array P840ar with 12Gb/s for each connected SAS SSD
  • The network controller is a HPE IB FDR/EN 40Gb 2P 544+FLR-QSFP Adptr

Thanks a lot!

silent

Posted 2016-11-21T12:28:56.363

Reputation: 101

Probably every protocol is able to do so, assuming you have the hardware to back it up (encrypt/decrypt the data fast enough). Depending on what you use there could special hardware like SSL accelerators/offloaders available. If you're using SSH why would you encrypt it again? That would already be an encrypted connection. As to quote from the project page: It is designed to be a reliable "back-end" tool that can be used directly or easily driven by other programs and scripts. – Seth – 2016-11-21T12:58:20.773

Don't you think the answer depends on the hardware available? It is not only a matter of encryption (though that does matter), it is also a question of how fast you can read and write to disks: at 40Gb/s, your data channel will saturate even the most recent SATA3.2 technology. So it is impossible to answer unless you specify the hardware available to you. – MariusMatutiae – 2016-11-21T13:27:03.770

@Seth The SSH tunnel was only an example if the actual transfer would be unencrypted, such as netcat. If e.g. SFTP would be fast enough, then of course I would not use SSH underneath. – silent – 2016-11-21T13:34:37.717

@MariusMatutiae As I said, the storage should be able to provide the data (we are talking about SAS SSDs each on a dedicated 12GB/s link to the controller). It's a HPE Smart Array P840ar to be precise. The network hardware is an HPE IB FDR/EN 40Gb 2P 544+FLR-QSFP Adptr) – silent – 2016-11-21T13:37:08.940

The fastest way would be the one with the least overhead so you'd have to think about what you really want to do here. As an example you could use plain netcat and encrypt the files beforehand. So the network transfer would be fast but you'd need preparation time. You could use a SSH or IPSec connection and use a connection based encryption. In addition you will have to think about the structure of the data. It could be worth it to zip them or at least use a tar to get a consecutive/monolithic file rather than a lot of small files (could be important if your solution is FTP based). – Seth – 2016-11-22T07:12:54.327

@Seth I already mentioned the file sizes above. They are probably already zipped – silent – 2016-11-22T11:57:22.447

If all those things are clear and you won't go into any more detail or have any more restrictions, you already have all the answers in the question you gave as a references. The thing left to do is actually write a script that does the job. The second reference has a non encrypted netcat example and the first one has a SSH/SCP example. That way you already have a two ready to go approaches and just need to substitute your files. Depending on the application your data is from there might still be a smarter way to go about it like using a cluster with replication. – Seth – 2016-11-22T13:30:27.697

No answers

Asked: 2016-11-21T12:28:56.363

Viewed: 302 times

Active: 2016-11-21T13:41:29.693