0
I am using cURL to download many files and concatenate them to STDOUT. About 100,000 small files. I would like to see progress against the 100,000. Is this possible with curl or by using curl into xargs?
Interested in only standard command line solutions.
William Entriken
Posted 2012-12-14T12:38:46.100
Reputation: 2 014
1
There are a few different things you could do here: but without knowing exactly your curl methodology I can just offer a few suggestions.
Make an iterative counting for loop:
for file {1..100000}; do echo "downloading: $file" >&2 ; curl [whatever] ; done
This will redirect the "downloading: $file"to STDERR so if you're using a redirect on STDOUT it won't mess it it up but you can still see it on the screen
for file {1..100000}; do echo "downloading: $file" >> progress.out ; curl [whatever] ; done
This writes progress to a file so it won't show up on the screen at all, then you can just tail -f progress.out
If your curl isn't iterative:
terminal 1:
curl [whatever] | tee progress.out
terminal 2:
watch -n5 "grep -c '[unique file delimiter]' progress.out"
tee writes a copy of STDOUT to a file, and you grep count for something that only appears once per HTML doc... maybe a <HEAD> or <HTML> tag or something else. Watch will just run the grep it every 5 seconds to let you know how many you have completed.
MattPark
Posted 2012-12-14T12:38:46.100
Reputation: 1 061
0
GNU Parallel is more or less standard these days:
cat urls | parallel -j30 --eta curl ... > out
Added benefit: multiple curls will be run in parallel.
Watch the intro videos to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Ole Tange
Posted 2012-12-14T12:38:46.100
Reputation: 3 034
it is possible to display something like: "downloading [200/100000]". but as you said, you want to print the file contents to stdout. if you got status/progress information, they should be printed to stdout as well. it will mess your stdout up. what do you exactly want? – None – 2012-12-14T14:37:21.233