1

I am trying to download a large file which is .tar.gz and (> 2GB) in size. The bash script also does lots of other things, so I want to kick off the download and then continue processing other commands in the bash script. I am using wget and piping to tar and running it in the background with &.

wget -q "https://download-of-large-file-here.tar.gz" | tar xz &
# other commands here

However, this is breaking with:

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now

Any idea how to make this work?

Justin
  • 5,008
  • 19
  • 58
  • 82

1 Answers1

7

You will have to instruct wget to write to STDOUT, you do this by passing -O - to wget, otherwise it will save the file to disk. tar is failing because as executed it's expecting something from STDIN.

This should work:

wget -q -O - "https://download-of-large-file-here.tar.gz" | tar xz &

Where -q suppresses the default progress information from STDOUT and -O - sets the output file to STDOUT.

Brandon DeRosier
  • 246
  • 1
  • 2
  • 8
  • This works, but the `wget` and `tar` process never seem to terminate. See a screenshot from `htop`. http://i.imgur.com/xwHmg8D.png – Justin Aug 12 '15 at 20:01
  • Does `tar` ever finish extracting? What happens when you run the commands separately? (e.g. `wget -O "test.tar.gz" "https://download-of-large-file-here.tar.gz"`, then `tar xz test.tar.gz`) – Brandon DeRosier Aug 12 '15 at 20:58