1
I am using curl through Cygwin on Windows 7 to download content from Apache Solr in CSV format. This is my command which is running in a for loop
curl -o ret-key-data-$i.csv $URL
As the content is very large, I am downloading 100k rows at a time. The file size is approx. 4mb each. However, if I increase the row count to 1 million, curl is not writing any content to disk.
What other option do I have to set to ensure curl can handle larger content?
EDIT: The command gives no error. It does not hang. It waits for a period of time and ends successfully but generates a zero kb file. Looks like the network is timing out.
cosmos
Posted 2014-05-12T05:36:46.760
Reputation: 111
I use cURL on a daily basis to download files on the hundreds of GB's and have not had an issue with limits to write to file. Can you post the results using the verbose switch
-v? It would be helpful to see the details of the request and the response from the server. – MikeBoss – 2017-03-14T21:27:02.137Please be more specific as to what happens. How is it "not writing any content"? Does the command wait forever and there is no output file, or an empty one? Do you see any messages output to terminal? – slhck – 2014-05-12T05:52:36.947
could the web server be timing out and severing the connection? – rob – 2014-05-12T08:05:27.573