Parallel Download Processes

Question

Using a Bash terminal, how can I have 5 ± 2 concurrent cURL processes downloading at any time in a script, until there aren't any more links?

Pseudocode:

Links = {a.txt, b.txt, c.txt, d.txt ... x.txt, y.txt, z.txt}

Loop URLS in Links
    if (less than 5 cURL processes)
        cURL URL
    else
        wait until 1 cURL process stopped

Answer 1

Try this as a starting point (its untested)

#! /bin/bash
# LINKS is a list of space separated links use text below, otherwise
# replace first uncommented line below with
# for each in `cat $LINKS`

for each in $LINKS
do 
    count=`ps waux | grep "curl" | wc -l`
    until [ $count -gt 4 ]
    do
        count=`ps | grep "curl" | wc -l`
        sleep 1
    done

    curl $each &
    fi

done