Best way to transfer files across unstable LAN?

Question

This is very similar to Question 326211, but in this case, the LAN is an unstable Wi-Fi connection.

I need to transfer about 11 GiB of files between two computers, both running Linux (although one may be rebooted into Windows.) Their connection is both slow and unstable (due to Linux's awful Wi-Fi support,) but removable media (such as a flash drive or external hard drive) is not an option at this time.

Right now, I'm slowly transferring the files, one by one, across SFTP, but I have to reconnect each computer approximately every 90 seconds, and the computers are not very close to each other, so this is not feasible.

This is not a duplicate of Question 30186; that one specifically concerns Windows 7, and all the proposed solutions involve closed-source, Windows-only programs (which are all spyware IMHO, and are all off the table even if I trusted them - one of the computers is Linux-only.)

Answer 1

What about using something like 7zip or winrar to create split archives of, say 50MB each. That way you can transfer files until connection drops but you don't need to start from scratch each time.

# -mx0 indicates no compression to be used (quickest method)
7z a -mx0 -v50m video.7z myVideo.mp4

Once all files transferred (using scp or equivalent) just recombine and extract? You'll probably also get better speeds from transferring multiple files at once.

--

EDIT: I know you said removable media was out of the question, but could you remove the hard disk from computer 1 and just plug into computer 2 at all to perform the transfer?

Answer 2

First of all I'd try and see whether it's possible to make the connection more stable. File transfer apart, I don't think it's healthy to work with a connection that drops every 90 seconds.

A simple USB WiFi dongle could work miracles there (first, investigate which device is actually causing the disconnection: host A, host B, or the access point?

Possibly, Linux has nothing at all to do with the signal loss, but's rather a matter of WiFi instability per se. Try locking the access point to a lower speed. Even a paltry 11 Mbps, if rock-solid, will outperform a flaky 54 Mbps connection. And very often, cards will try for the higher speed even if, given the situation, they can't possibly deliver.

If you're stuck with a connection that drops and renegotiates every 90 seconds, your best bet might be either rsync or chunk the files into small enough pieces to filter between renegotiations:

split -b 10000000 file.mp4

will split the file into N chunks of (say) 10 MiB each, called xaa, xab, xac... Once the copy is complete, you can reassemble the chunks on the other side using cat.

Another possibility could be to abandon TCP altogether and use an UDP based protocol, for example UDT or UFTP.

Finally, you could set up a simple Web server on one machine, and use wget or any HTTP client supporting retrieval integrity, byte-range and auto-retry to download the whole set of files (which, if they aren't already, it would be better to compress).

This is longer to set up, but then it ought to go on automatically and uses available and tested & proven software (apache, wget).

Come again - gone again - Finnegan

Granted that you need to be sure the scripts are synced, or you'll waste a lot of connection time, you can probably play with either iwconfig or ifup/ifdown in order to force the access point to behave:

while true; do ifdown wlan0; ifup wlan0; sleep 90; done

or if you want to run it as a script (it will require root access either way, to run if* commands):

#!/bin/sh
while true; do
    echo "Going down..."
    ifdown wlan0
    echo "Coming up..."
    ifup wlan0
    echo "Ready"
    sleep 90
done

A (hopefully) better way

The problem with the "sleep of 90 seconds" is that unless something is really weirdly flaming wrong in the router, the time it stays up won't be exactly 90 seconds. So we're wasting a lot of time either staying up when the WiFi carrier is down, or going down when the WiFi carrier might have stayed up.

So let's say that the router has IP 192.168.0.1, on both Linux hosts we run

#!/bin/sh

IP=192.168.0.1

while true; do
    if ( ping -f -c 3 -w 1 $IP | grep "0 received" > /dev/null ); then
        ifdown wlan0
        ifup wlan0
        # Extra time: we MUST be sure the network is up!
        sleep 5
    fi
    sleep 2
done

This script will shoot three packets towards the router. The packets are quite small and won't greatly affect the bandwidth. One dropped packet might be happenstance, two a coincidence -- three dropped packets trigger a WLAN power cycle. Hopefully, this should waste no more than at most two seconds' time per cycle (instead of up to 90 seconds), and never trigger a power cycle unless there's a need.

It goes without saying that the PING functionality must be verified on the router - if the router doesn't answer pings, or drops ICMP when under TCP/UDP load (e.g. because of protocol prioritization), then the second version of the script could do far more harm than good.

But now that I think of it, did you try and fiddle with iwconfig parameters? For example the sensitivity threshold:

sens Set the sensitivity threshold. This define how sensitive is the card to poor operating conditions (low signal, interference). Positive values are assumed to be the raw value used by the hardware or a percentage, negative values are assumed to be dBm.

Depending on the hardware implementation, this parameter may control various functions.

On modern cards, this parameter usually control handover/roaming threshold, the lowest signal level for which the hardware remains associated with the current Access Point. When the signal level goes below this threshold the card starts looking for a new/better Access Point. Some cards may use the number of missed beacons to trigger this. For high density of Access Points, a higher threshold make sure the card is always associated with the best AP, for low density of APs, a lower threshold minimise the number of failed handoffs. On more ancient card this parameter usually controls the defer threshold, the lowest signal level for which the hardware considers the channel busy. Signal levels above this threshold make the hardware inhibits its own transmission whereas signals weaker than this are ignored and the hardware is free to transmit. This is usually strongly linked to the receive threshold, the lowest signal level for which the hardware attempts packet reception. Proper setting of these thresholds prevent the card to waste time on background noise while still receiving weak transmissions. Modern designs seems to control those thresholds automatically.

Example :

   iwconfig eth0 sens -80
   iwconfig eth0 sens 2

Answer 3

while ! rsync \
  --bwlimit <KB/s value> \
  -rP \
  /path/to/directory_that_contain_the_data_to_be_transferred \
  [e-mail]:/path/to/target_directory ; \
  do sleep 90;
done

taken from https://patrick-nagel.net/blog/archives/434

You'd want to put this in a bash shell script. something like the following

#!/bin/bash

while ! rsync \
  --bwlimit <KB/s value> \
  -rP \
  /path/to/directory_that_contain_the_data_to_be_transferred \
  [e-mail]:/path/to/target_directory; \
  do sleep 90;
done

you need to setup ssh keys which is another question if you don't know how to do that but it's basically:

ssh-keygen -t rsa

that will create your public/private key pair. Copy your public key to the destination machine so you don't have to use password credentials each time the connection drops.

Answer 4

A lot of the other answers involve splitting the file into pieces and adding integrity checks - this is rather similar to how torrents work. Could you use something like rtorrent?

http://libtorrent.rakshasa.no/

Answer 5

Bittorrent Sync works across LAN. Current requirements (per manual) are:

Linux with kernel 2.6.16 (glibs 2.4) or newer on ARM/PPC/i386/x86_64