how to pipe data to sftp connection?

Question

ftp supports the put "|..." "remote-file.name" command to pipe data to an ftp connection. Is there something similar available for sftp?

In sftp i get the following error:

sftp 'jmw@backupsrv:/uploads'
sftp> put "| tar -cx /storage" "backup-2012-06-19--17-51.tgz"
stat | tar -cv /storage: No such file or directory

as above the sftp client doesn't obviously execute the command.

i want to use the pipe command to directly redirect the file stream to sftp. (because there is not enough space left to create a backup file on the same disk before uploading it to sftp server.)

I thought about using a FIFO for this, but neither SFTP nor SCP will read from a FIFO. — Tom Anderson, Jul 05 '12 at 12:27
also good idea, this is mine: http://www.qxs.ch/2012/07/05/sftp-upload-tool/ — JMW, Jul 05 '12 at 14:41

score 6 · Answer 1 · answered Oct 05 '18 at 13:32

As this is the first result you find by googling for this question and it hasn't been mentioned already I'll add the solution I found here as well:

you can use curls sftp implementation for this. As curl is probably already installed on a lot of systems this might be preferred to the solution using custom clients.

example usage:

pg_dump -d database | pigz -1 | curl -u username -T - sftp://sftpserver/folder/dbbackup.sql.gz

curl uses your .ssh/known_hosts file for key verification. This might fail in case your ssh client does use newer encryption standards not supported by the library used in curl

to fix this you can add the other key types to the known hosts file using the following command:

ssh-keyscan sftpserver >> ~/.ssh/known_hosts

or you can disable key verification using the -k flag (I wouldn't recommend that though)

Kenny Rasschaert · Answer 2 · 2013-03-25T12:51:13.397

I had a lot of fun figuring out a solution to this problem. It requires the tool nc (netcat) on both machines, and SSH (SFTP isn't needed).

In this example, I shall call the machine that has the data that needs to backed up linux-a, and the machine that needs to receive the backup linux-b.

On linux-a, have netcat listen on a port (I took 2000) and redirect it to a file. This will just sit there and wait until something comes through on that port.

[kenny@linux-b /var/backups]$ nc -l 2000 > backup.tgz

On linux-b, open up an ssh tunnel to linux-a, I used port 2000 again. This will redirect anything you throw at TCP port 2000 on localhost to TCP port 2000 on linux-a, where netcat is listening.

[kenny@linux-a /var/data]$ ssh -L 2000:localhost:2000 -CfN linux-b

Now create the tar archive, but send the output to stdout (using -) and pipe it to gzip for some compression. Now pipe that to another netcat that sends it to localhost on TCP on port 2000.

[kenny@linux-a /var/data]$ tar cf - important-data | gzip -fc | nc localhost 2000

We're done! On linux-b the netcat is no longer listening, and a new file is created. The best part is that the tar archive was never placed on the hard disk of linux-a.

[kenny@linux-b /var/backups]$ file backup.tgz 
backup.tgz: gzip compressed data, from Unix, last modified: Thu Jul  5 13:48:03 2012

I know it's not exactly what you asked for in the question, but if you have netcat available, its a viable solution to your type of problem.

Edit: I forgot about one thing: if you follow these instructions, you'll still have an SSH tunnel floating around on linux-a. Find out what the process ID is and kill it.

[kenny@linux-a /var/data]$ ps -ef | grep "ssh -L"
kenny     5741     1  0 13:40 ?        00:00:00 ssh -L 2000:localhost:2000 -CfN linux-b
kenny     5940  3360  0 14:13 pts/1    00:00:00 grep --color=auto ssh -L
[kenny@linux-a /var/data]$ kill 5741

i like it, but there are 2 problems: sftponly users usually cannot use port forwarding and sftponly users cannot access nc on the sftp server (remember they do not have ssh access) — JMW, Jul 05 '12 at 12:23
Off-topic, but why is there a screen shot of the "The Prestige" embedded in this post? — Rilindo, Jul 05 '12 at 14:10

score 2 · Answer 3 · answered Jun 19 '12 at 16:42

2

output-stream-generating-command | ssh user@remotehost 'input-stream-accepting-command' is an option, if your remote user has a valid shell.

answered Jun 19 '12 at 16:42

voretaq7

79,345
17
128
213

6

This is _NOT_ sftp! – jirib Jun 19 '12 at 18:23
3

@JiriXichtkniha No, it isn't, however SFTP is almost always implemented as a subsystem of your SSH server, and this is the closest analog to the functionality I see being requested from the question (pipe data through SFTP to be handled by a program on the other end). Sometimes the answer is "You're using the wrong tool - do `this` instead." - this strikes me as one of those times. – voretaq7 Jun 19 '12 at 18:31
2

sftp is protol, and is different from just piping via ssh, that's it! Your suggestion would not work if remote part would be SFTP only. – jirib Jun 19 '12 at 19:03
@JiriXichtkniha Hence the caveats in my answer. If you have a better alternative for them which works for a pure SFTP server that cannot handle SSH (or where the user is disallowed login, doesn't have a valid shell, etc.) by all means please post an answer with that solution - I would be interested in seeing it myself. – voretaq7 Jun 19 '12 at 19:32
2

i'm not allowed to login as ssh user – JMW Jun 22 '12 at 07:23
1

@JMW in that case as far as I know you're pretty well screwed here - I've never seen a version of sftp that supports piping to a command on the remote system (probably because it would defeat the `S`ecure bit by letting someone in your situation who doesn't have a shell run programs anyway). I could be wrong though - I'm mostly familiar with the SFTP functionality in OpenSSH and the proprietary SSH server Sun used to ship... – voretaq7 Jun 22 '12 at 16:00
you're right the sftp client doesn't support this - but stay tuned for the solution... :-) – JMW Jul 05 '12 at 11:05
i've added a sftp_stdin_upload and sftp_stdout_download client :-) – JMW Jul 05 '12 at 14:30

score 1 · Accepted Answer · edited Jul 28 '22 at 22:45

1

voretaq7 pointed out, that the sftp client doesn't support piped data transfer for users, that are allowed to use sftp only to connect to the server.

fortunately, there is libssh2, that supports sftp. so we just need 2 other clients using libssh2, that i've called:

sftp_stdin_upload (to upload to a sftp server)
sftp_stdout_download (to download from a sftp server)

the source code can be found under the following URL: http://www.qxs.ch/2012/07/05/sftp-upload-tool/

since i'm not that experienced in libssh2 programming, i'm happy for any feedback to the source code.

edited Jul 28 '22 at 22:45

sysadmin1138

131,083
18
173
296

answered Jul 05 '12 at 11:18

JMW

1,451
4
19
27

Doesn't work for me :-( /usr/bin/ld: libssh2-1.4.2/src/.libs/libssh2.a(knownhost.o): undefined reference to symbol 'EVP_sha1' /usr/lib/libcrypto.so.1.0.0: error adding symbols: DSO missing from command line collect2: error: ld returned 1 exit status – tobixen Jun 05 '15 at 09:50
This code has just a POC status. Feel free to fix it on your own. :-) – JMW Jun 14 '15 at 20:45
Uploading using libssh and libssh2 is quite slow in comparison to using `sftp` command-line client... Also, the link (to qxs.ch) returns a 404 today... – Mikhail T. Jul 28 '22 at 21:10

how to pipe data to sftp connection?

4 Answers4