4
0
I need to download 30000+ .mp3 files! each of them are ~200 kb..
I tried to download all of them or in 6 sets (5000 files each time) with Internet Download Manager but it freezes.. Which download manager could handle this much files?
Omar
Posted 2011-10-07T22:48:59.423
Reputation: 85
4
I routinely do 3k <.5MB files, and I use Free Download Manager.
I can't say if there is better or more light weight solutions out there, but, this works and does what I need - I go page by page, right click, add all to IDM, then when I finished, I highlight them all in IDM and do start!
I have never done 30k all in one go, but, I hope this helps!
William Hilsum
Posted 2011-10-07T22:48:59.423
Reputation: 111 572
1FDM is a good general purpose download manager. 30k might be pushing it, though... – ta.speot.is – 2011-10-08T09:22:35.453
I tried FDM and its okay though it fatal-crashes a lot when I add the links, I tried to add 6000 links each time and its still crashing :( – Omar – 2011-10-08T21:10:53.117
1@Omar - Do you get impatient or do you leave it running? 6k is a hell of a lot for any program... possibly leave it running to get it's way through your list. – William Hilsum – 2011-10-08T21:20:20.773
1@WilliamHilsum Haha no I obviously leave it running for a long time though when I do anything after a looong time I get the fatal-crash dialog – Omar – 2011-10-08T21:23:38.220
@Omar - It depends what "fatal crash" dialog it is... if it is a single threaded application, the UI will be unresponsive and look like it has crashed - it will most likely come back to life after finishing... but, if you click on it, it will appear dead/unresponsive/crashed... If you have time to test, give it a 6k list and leave it overnight and see if it has worked by morning. If it hasn't I am out of ideas and clearly this program is not for you! – William Hilsum – 2011-10-08T21:26:57.550
Wget is much better. Use the mirror option. As I said, I can mirror it for you, but it will take some time. When I'm done, I'll post the 7z. – Apache – 2011-10-09T12:00:14.090
3
As @Shiki already said: wget
wget -i will read all URLs from external files and process them
Lazy Badger
Posted 2011-10-07T22:48:59.423
Reputation: 3 557
1
Jdownloader http://jdownloader.org/download/index Is an interesting, multi-platform Java downloader that is being developed continually. It is Not lightweight at all, and could be a security risk for all I know. 30k would probably have to be done in sections. but it has a way to backup the list, and load lists in. Its main design seems to be more about downloading from "public servers", but it works good for normal stuff.
I have seen 5000 items in it, but that was because I asked it to analise a link that went on forever, ooops.
Psycogeek
Posted 2011-10-07T22:48:59.423
Reputation: 8 067
1
I use curl (http://curl.haxx.se/). It has a bunch of good features for mass downloads. It allows you to specify many files on one invocation, like this:
CURL.EXE http://example.com/file[1-10].mp3
That will download "file1.mp3", "file2.mp3", and so on.
I've used it to download many files, but not 5,000. However, in principle it should work.
mauvedeity
Posted 2011-10-07T22:48:59.423
Reputation: 745
2Post the url, I'll get them with linux, 7z them (store archive) and upload it somewhere. If that's a solution. As an answer: Go with wget. If possible. The URL would help us a lot. – Apache – 2011-10-07T23:14:48.337
1Wow I feel like I'm asking too much. I've updated the post above with 2 links each containing 6300+ links. Thanks for the offer Shiki – Omar – 2011-10-08T21:16:05.623
1If you can manage it, ask someone on the other end to zip them up for you. You won't get much additional compression (if any), but they will transfer much faster as a unit. – Joel Coehoorn – 2011-10-08T21:23:39.563
1You are not asking too much, we just want to help. :) – Apache – 2011-10-08T21:33:22.307