How can I run the equivalent “curl” command that performs similarly to this “wget” command?

Question

I have a wget command:

wget -r --no-parent -nH -nd -q --show-progress --progress=bar:force ftp://ftp.ncdc.noaa.gov/pub/data/swdi/database-csv/v2/

But it only works in newer versions of wget. Unfortunately I have an older version of wget (GNU Wget 1.15 built on darwin14.3.0) and can’t upgrade it. Specifically it’s the wget --show-progress option that isn’t recognized.

I am wondering if there is an equivalent command for curl? The version of curl on my system is 7.37.1 (x86_64-apple-darwin14.0).

Answer 1

But it only works in newer versions of wget. Unfortunately I have an older version of wget (GNU Wget 1.15 built on darwin14.3.0) and can’t upgrade it. Specifically it’s the wget --show-progress option that isn’t recognized.

Based on what I know about how curl works and how wget works, they might superficially seem the same but there is no equivalent of wget recursive functionality in curl. To do something like that in curl requires some fancy Bash scripting to wrap around the curl command.

So when you say you can’t upgrade wget, what exactly do you mean? Since wget is a GNU tool, it’s not installed with Darwin (which is BSD-based) and is often compiled from source. In fact the version details you have provided is clearly compiled from source in some way since the version is “GNU Wget 1.15 built on darwin14.3.0.” So if that version of wget was installed by something like Homebrew, there is hope since if Homebrew is on your system then you should have Xcode—and related command line tools/compilers—installed on your system as well.

Now knowing that, you should be able to compile an updated version of wget—such as 1.16.3—from source; bypassing the Homebrew installed version and just having it executable from your home directory. You don’t have root/sudo access to do this; here’s how you can do it.

Compiling wget from source.

First grab a compressed archive from an official wget source site:

curl -O http://ftp.gnu.org/gnu/wget/wget-1.16.3.tar.gz

Next, decompress the archive like this:

tar -xzf wget-1.*.tar.gz

Now go into the decompressed directory:

cd wget-1.*

Run this configure command:

./configure --with-ssl=openssl

If somehow that configure command fails, you might need to add a --with-libssl-prefixvalue like this:

./configure --with-ssl=openssl --with-libssl-prefix=/usr/lib

Once the configure process completes, run make:

make

If that runs cleanly a newly compiled version of the wget 1.16.3 binary is ready for you to use.

Running the freshly compiled version of wget from your home directory.

Now at this point, most instructions would say you should run sudo make install to get wget fully installed on the system. But if you don’t have root/sudo privileges—or you don’t want to upgrade the wget version that is installed system-wide—you can still run the wget binary you just compiled like this. First, let’s back out of your wget source directory like this:

cd ~/

Now just run the wget 1.16.3 binary you just compiled explicitly like this to confirm it’s working:

~/wget-1.16.3/src/wget --version

The output should be something like this:

GNU Wget 1.16.3 built on darwin13.4.0.

+digest +https +ipv6 -iri +large-file -nls +ntlm +opie -psl +ssl/openssl 

Wgetrc: 
    /usr/local/etc/wgetrc (system)
Compile: 
    gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/usr/local/etc/wgetrc" 
    -DLOCALEDIR="/usr/local/share/locale" -I. -I../lib -I../lib 
    -DNDEBUG 
Link: 
    gcc -DNDEBUG -liconv -lssl -lcrypto -ldl -lz ftp-opie.o openssl.o 
    http-ntlm.o ../lib/libgnu.a 

Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Originally written by Hrvoje Niksic <[e-mail]>.
Please send bug reports and questions to <[e-mail]>.

Which means you are now in business!

Running your wget command from your home directory.

So you can now run the wget command posted in your question like this:

~/wget-1.16.3/src/wget -r --no-parent -nH -nd -q --show-progress --progress=bar:force ftp://ftp.ncdc.noaa.gov/pub/data/swdi/database-csv/v2/

And a small tweak—but one worth noting—is that many wget options have full names and shortened acronyms. So --no-parent can be shortened to -np to make the command look like this:

~/wget-1.16.3/src/wget -r -np -nH -nd -q --show-progress --progress=bar:force ftp://ftp.ncdc.noaa.gov/pub/data/swdi/database-csv/v2/

Post-usage cleanup.

And if you want to get rid of the wget 1.16.3 stuff you just compiled—since you say the computer you are working on is not yours—then just delete the wget source files you just worked with like this:

rm wget-1.16.3.tar.gz
rm -rf wget-1.16.3