2
2
Are there small and simple file transfer tools for Linux, which can use "zero-copy" style of transferring files from HDD (SSD) to tcp socket, and back?
I want to often send huge amount of uncompressible unique data (tens-hundreds of GB) with Ethernet, 1 Gbit or may be 10 Gbit, between two PCs. I need no encryption, nor compression, no deduplication/delta transfers and no any extra checksumming from the tool (so, no scp and no default rsync); just high bandwidth of linear disk reads and high network utilization.
And also I want the tool not to make lot of memory copies, but use "zero-copy" infrastructure of modern Linux kernel: syscalls splice, vmsplice, sendfile, etc. With zero-copy tool can skip most data copying procedures, instructing kernel to read some data from HDD/SSD/RAID (DMA read to pagecache) and then repack it into packets and to program network card to doing DMA directly from pagecache. This "zerocopy" approach will lower system load from the transfer, because CPU will not have to crumble up every byte of file data. One side of transfer may have slow CPU...
osgx
Posted 2014-05-28T20:35:37.543
Reputation: 5 419
Apache has sendfile support.
– Cristian Ciupitu – 2014-05-28T20:47:44.850This question might be better suited for Software Recommendations.
– Cristian Ciupitu – 2014-05-28T20:51:59.603Cristian Ciupitu, Apache is huge daemon. I searching for something small and simple, like rsync+rsyncd/scp+ssd/netcat... – osgx – 2014-05-28T21:03:47.757
1
Interesting thread from fa.linux.kernel: https://groups.google.com/forum/#%21topic/fa.linux.kernel/0Gw6MIvix_k "splice/vmsplice performance test results" by Jim Schutt, 2006-11-16 with source code of "dnd" single-file disk-network-disk transfer tool. It have several methods of data reading/writing/splicing.
– osgx – 2014-05-29T01:12:05.950Have you seen this article? http://blog.superpat.com/2010/06/01/zero-copy-in-linux-with-sendfile-and-splice/comment-page-1/
– MariusMatutiae – 2014-05-30T07:37:33.170