How can I download a subpart of file in linux/windows?

I want to download a huge file (dataset) which is about 200GB but do not have enough disk space in my internet-connected PC. I want to download a piece of the file (e.g. 30GB chunks) in each run and move the chunks to my server (which has large storage capacity). Then I will eventually merge the chunks to build the whole file. (FYI, the server is network-isolated, I transfer the files using a flash memory.)

How can I download a part of a file (for example, from 10,000,000'th byte to 20,000,000'th byte)?

Any tool/command in linux (preferred) or windows is acknowledged.

Thanks

Ali

Posted 2014-04-15T19:56:04.447

Reputation: 159

How are you going to move the 30 GB chunks to your server? Are these two computers air-gapped? If not, can't you just set up a (temporary) connection so that you can download directly to the server? – Jan Doggen – 2014-04-15T20:45:17.780

Yes, the server is isolated by the admin, but I can transfer the files to the server using a flash memory. – Ali – 2014-04-15T20:48:26.427

Answers

You can use cURL for this. Positions are specified in bytes:

curl -r 10000-20000 http://server/file >chunk2

to get a second 10kB chunk.

liori

Posted 2014-04-15T19:56:04.447

Reputation: 3 044

While this is probably the best answer to your question, the best solution to your problem would probably be an USB hard drive connected to the internet-connected PC. – Squeezy – 2014-04-15T21:56:31.470

In order for you to download a file in such a manner it would be easier if the person providing the file broke it into chunks for you, then you use the tool they broke it up with to put it back together.

In order for you to break it into chunks reliably, you'd need the entire file anyhow, which you cannot get.

eyoung100

Posted 2014-04-15T19:56:04.447

Reputation: 419

I Know, but they have shared the file in this way! this is MY problem anyway!!! – Ali – 2014-04-15T20:15:07.387