75

Rsync over ssh, works great every time.

However, trying to rsync to a host which allows only sftp logins, but not ssh logins, provides the following error:

rsync -av /source ssh user@remotehost:/target/

protocol version mismatch -- is your shell clean? (see the rsync man page for an explanation) rsync error: protocol incompatibility (code 2) at compat.c(171) [sender=3.0.6]

Here's the relevant section from the rsync man page:

This message is usually caused by your startup scripts or remote shell facility producing unwanted garbage on the stream that rsync is using for its transport. The way to diagnose this problem is to run your remote shell like this:

          ssh remotehost /bin/true > out.dat

then look at out.dat. If everything is working correctly then out.dat should be a zero length file. If you are getting the above error from rsync then you will probably find that out.dat contains some text or data. Look at the contents and try to work out what is producing it. The most com‐ mon cause is incorrectly configured shell startup scripts (such as .cshrc or .profile) that contain output statements for non-interactive logins.

Trying this on my system produced the following in out.dat:

ssh-dummy-shell: Command not allowed.

As I thought, the host is not allowing ssh logins.

The following link shows that it is possible to accomplish this task using fuse with sshfs - however it is extremely slow, and not fit for production use.

Is there any chance of getting rsync sftp to work?

Tom Feiner
  • 16,758
  • 8
  • 29
  • 24
  • 3
    lftp will do the trick. I will sync local to remote with the -R switch –  May 28 '13 at 17:09
  • Do you know how the commands are restricted on the server? If it is by setting ForceCommand in sshd_config, then see if you can modify the file that points to. Or if it's set by command in authorized_keys, then maybe you can access authorized_keys over sshfs and change it? – ptman Apr 02 '14 at 10:56
  • You could try [csync](https://www.csync.org/): `csync /home/csync sftp://csync@krikkit.galaxy.site:2222/home/csync` See this [answer](http://superuser.com/a/6897/216090). – nachtigall Sep 06 '14 at 04:45
  • 8
    Eventually, I did it with [lftp](http://lftp.yar.ru/): `lftp -e "mirror -eRv /local/dir/ /remote/dir; quit;" sftp://user@server:port` – nachtigall Sep 06 '14 at 05:51
  • @nachtigall do you know how to use lftp with multiple input files / a list of files and dirs? – bortran Apr 04 '16 at 14:10

8 Answers8

43

Unfortunately not directly. rsync requires a clean link with a shell that will allow it to start the remote copy of rsync, when run this way.

If you have some way of running long-lived listening processes on the host you could try starting rsync manually listening for connections on a non-privileged port, but most techniques for doing that would require proper shell access via SSH too, and it relies on the hosts firewall arrangements letting connections in on the port you chose (and the host having rsync installed in the first place). Running rsync as a publicly addressable service (rather than indirectly via SSH or similar) is not generally recommended for non-public data though.

If you host allows scripting in PHP or similar and does not have it locked down so extra processes can not be execed by user scripts, then you could try starting rsync in listening mode that way. If your end is connectible (you are running SSH accessible to the outside world) you could try this in reverse - have a script run rsync on the server but instead of listening for incoming connections have it contact your local service and sync that way. This still relies on rsync actually being installed on the host which is not a given, or that you can upload a working copy, but does not have the security implications of running an rsync daemon in a publicly addressable fashion and talking to it over an unencrypted channel.

Messing around as described above may be against the hosts policies though, even if it works at all, and could get you kicked off. You are better off asking if a full shell can be enabled for that account and either abandoning rsync for that host or abandoning that host and moving elsewhere if they will not do that.

David Spillett
  • 22,534
  • 42
  • 66
  • 2
    Note that it doesn't have to be a full shell; it just has to allow the rsync command with the proper arguments to run. scponly is useful for this. – sciurus Sep 20 '11 at 14:15
27

An alternative to using rsync is to instead use lftp (which can connect to sftp) and use the mirror command. For example one can do

lftp
~> open -u user,password sftp://host.com
~> mirror remotedir outdir
~> quit
xioxox
  • 539
  • 5
  • 9
23

Theoretically, yes. You can mount the remote filesystem on your local machine using FUSE. Then you can run a local copy of rsync between the mounted directory and the local directory. I've not personally tried this, but it should work in theory. It would likely be a lot less efficient that performing the rsync over SSH because it would need to transfer at least part of each file to perform the comparison.

Kamil Kisiel
  • 11,946
  • 7
  • 46
  • 68
  • 14
    That should work when timestamp+size comparison is enough to decide what needs to be send. As soon a a checksum is needed the entire remote file will be read over the line as well as the updates sent, so make sure you have rsync's options set such that it does all-or-nothing for each file (not using --ignore-times or --checksum and having --whole-file specified may be enough). Some options like --link-dest are highly unlikely to work properly (or at all) this way. – David Spillett Apr 26 '10 at 01:11
  • Thanks for adding some details David. That's the kind of limitations in efficiency I was thinking of, just didn't articulate them quite as well :) – Kamil Kisiel Apr 26 '10 at 17:42
  • It's a workaround with limitations, yes, but it is good enough if there is no way to invoke "proper" rsync – Valery Lourie Jul 13 '14 at 09:15
6

Instead of Rsync, rclone supports syncing from/to and mounting SFTP network shares.

  1. Install rclone
  2. rclone config and follow the steps to add a new remote endpoint configuration. Input sftp at the protocol backend selection.
  3. rclone sync local_path remote:remote_path (docs)

IMO, rclone provides a more automation-friendly CLI than LFTP.

I know this does not answer the original question but probably the needs of some people which come here.

ominug
  • 664
  • 7
  • 5
6

a bit late, but here is how I do it, using sshfs

  source /scratch/slimdata/password.sh
  mkdir tmp_mnt
  echo $PASSWORD | sshfs user@host:dir tmp_mnt -o password_stdin
  rsync -rutL --delete tmp_mnt/ to_sync/
  fusermount -u tmp_mnt
  rmdir tmp_mnt
irri
  • 3
  • 2
jonathan
  • 71
  • 1
  • 1
  • 4
    As @David Spillett says in his comment on an earlier answer (http://serverfault.com/questions/135618/is-it-possible-to-use-rsync-over-sftp-without-an-ssh-shell/135649#135649), this will work but is a bit of an inefficient use of the rsync protocol. – Evan Anderson Sep 20 '11 at 12:41
  • 3
    jonathan, I think your solution is wrong. If you mount a remote drive and execute rsync like you would sync two local folders, there is always transfered everything on syncing since the rsync process on your local machine calculates the hash values of the remote files. This means the whole file is transfered and the benefits of rsync are lost. Edit: ah this answer has been given several times... sry for replication :D – Thekwasti Mar 22 '12 at 10:20
5

Option combining best of all

It appears to have the following advantages that other answers here don't have:

  • fully benefits from SSH (secure)
  • fully benefits from rsync (bandwidth-efficient protocol, all rsync options like bandwidth limitation)
  • actually efficient (unlike sshfs which save the days sometimes but is still slow in practice)
  • doesn't need arbitrary shell commands server-side
  • doesn't need server to allow ssh tunnels
  • doesn't need server to run a rsync daemon

I have not yet tested it, but I've successfully used all those features except rrsync.

How to do it

  • create a key locally with ssh-keygen (openSSH feature)
  • allow login only with that specific key, with an entry in server's ~/.ssh/authorized_keys (openSSH feature)
  • associate this key to a specific command, with ~/.ssh/authorized_keys (openSSH feature) using as command the script rrsync distributed with rsync, something like command="$HOME/bin/rrsync -ro ~/backups/",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-

Then you can rsync from client normally.

If you need more details

Restricting SSH Access to rsync | Guy Rutenberg

One step beyond link above

The command at the end of that page can be made shorter. In ~/.ssh/config create a stanza like this:

Host remote # can be host or ip or custom-label User user # login on remote host HostName optional-dns-resolvable-host-or-ip # if label used above IdentityFile ~/.ssh/id_remote_backup

then your rsync command from client

rsync -e "ssh -i $HOME/.ssh/id_remote_backup" -av user@remote: etc2/

becomes

rsync -av user@remote: etc2/
Stéphane Gourichon
  • 714
  • 6
  • 13
  • 7
    Unfortunately this doesn't have anything to do with using rsync with an SFTP server though, this is about configuring an OpenSSH server to allow `rsync` to be run, sidestepping the SFTP aspect. If you are presented with someone else's SFTP server that you don't have any control over then this answer can't solve the problem. – Malvineous Sep 07 '18 at 06:14
  • You're right. I might have actually written this to answer another question and posted here by mistake. I'm considering removing my answer here. – Stéphane Gourichon Sep 07 '18 at 15:31
3

No. rsync works by running rsync on the other side and communicating with it, which means that some form of shell access is required.

Ignacio Vazquez-Abrams
  • 45,019
  • 5
  • 78
  • 84
2

An alternative would be to start rsync as daemon and connect to it through a SSH tunnel.

Dan Andreatta
  • 5,384
  • 2
  • 23
  • 14
  • This answer is too short to be completely helpful, but if the server side cannot allow shell access but can allow (1) a rsync server listening locally (not available from the outside, for security), and (2) ssh tunnel, then in principle one could log in to server via a rule in `authorized_keys` that allows a specific tunnel and runs e.g. `sleep 86400`. Then the client side could then `rsync` via the tunnel. – Stéphane Gourichon Jan 08 '18 at 09:46