cURL SFTP upload speed only 50% compared to command line SFTP

Question

I need to use cURL to upload some files to an SFTP server, it has to be using cURL because I'll be later using cURL_multi etc to transfer files simultaneously (unless anyone knows a command line library which supports FTP, SFTP using password, SFTP using key, and supports up to 10 simultaneous transfers).

The problem I have is cURL is running painfully slowly. Uploading using the standard SFTP client on Ubuntu 12.04, I'm getting around 70Kb/s (this is fine, that's just the speed of the server), but when using cURL, it's around 35Kb/s, half the speed of SFTP :/

Does anyone know of any ways I can speed up cURL, preferably without having to recompile cURL, but I will if I need to.

Here's the command for SFTP:

/usr/bin/sftp -oProtocol=2 -oIdentityFile=/path/to/key.pri -oPort=5000 -oStrictHostKeyChecking=no [e-mail]

then just put filename.wav when it's connected...

and with cURL:

curl -v -u username: --key /path/to/key.pri --pubkey /path/to/key.pub -T /path/to/filename.wav sftp://thedomain.com:5000/folder/TEST/filename.wav --insecure

This is the cURL version info:

curl 7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 libssh2/1.2.8 librtmp/2.3 Protocols: dict file ftp ftps gopher http https imap imaps ldap pop3 pop3s rtmp rtsp scp sftp smtp smtps telnet tftp Features: GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP

Surely it can't be this slow for everyone?

Answer 1

This is unfortunately a known issue with curl. It seems it can be mitigated by recompiling with a larger CURL_MAX_WRITE_SIZE; see this entry from their mailing list.

Since you're trying to achieve multi upload in the end, you may still succeed in filling your outgoing pipe with SFTP over curl, which would give you the best possible throughput anyway :)