How to download a file behind authentication via Wget or cURL

Question

Possible Duplicate:
Wget a page that requires logging in

I wish to download file from http://mbr.nlm.nih.gov/Download/MetaMapped_Medline/2011/. However, in order to download the file, I need to enter a password and username, which are accepted via an HTTP form.

Is there a way by which I can enter the password/username using the command line?

Answer 1

You can login with your browser (with preferable "Remember me" option), export browser cookie (for Firefox 3.x, here is a python code to do that) and use --load-cookies with wget.

Answer 2

wget and curl are only capable of handling basic HTTP authentication and not custom form based authentication.

What you will need to do is first examine the HTML of the login form to find the form elements that the login system uses and the form's action, then use curl or wget to send a request in the right format (POST or GET) to the form's action URL containing the login variables and data.

That request should return a cookie containing a session key or something similar. Submitting that cookie along with any future requests should then allow you to access the pages on the site that require you to be logged in.