How to download this webpage with Wget?

I want to download the webpage http://forum.ubuntu-it.org/, but it requires a username and password. So I have used this:

wget --save-cookies cookies.txt --post-data 'user=goyamy&passwrd=mypassword' http://forum.ubuntu-it.org/

But it does not work! Why?

xRobot

Posted 2010-11-25T02:03:42.770

Reputation: 361

Answers

That's possibly because the server uses session cookies to track authentication. Add the option --save-cookies alongside to force the cookie to be saved. So your commmand looks like this:

wget --keep-session-cookies --save-cookies cookies.txt --post-data 'user=goyamy&passwrd=mypassword' http://forum.ubuntu-it.org/

I haven't tested it though.

Kibet

Posted 2010-11-25T02:03:42.770

Reputation: 338

Here is an example script which will dump cookies from Chrome (v19).

#!/bin/bash -e
#
# Quick and dirty script which dumps all Chrome cookies in 
# the specified SQLite database to stdout in Netscape format.

COOKIE_FILE='~/.config/google-chrome/Default/Cookies'

echo -e '.mode tabs \n select host_key, httponly, path, secure, ' \
  'expires_utc/10000000, name, value from cookies;' |
  sqlite3 $COOKIE_FILE |
  sed -e 's/\t0\t/\tFALSE\t/g ' -e 's/\t1\t/\tTRUE\t/g'

user137906

Posted 2010-11-25T02:03:42.770

Reputation: 61

Please "quote" the variable in sqlite $COOKIE_FILE. – ankostis – 2015-06-16T16:18:03.247

It worked after replacing the ~ in COOKIE_FILE by its particular value in my configuration. I suppose there's a lacking export or the like in my system. The error that I received before making this change was unable to open database file. – naitoon – 2013-08-11T00:59:23.633

As Colin suggests, this site is using session cookies for authentication, but his answer won't fully work because it won't get you logged in.

You need to have a cookie for wget to pass to the server on the initial request. Use wget's --load-cookies option (documented here). Note that this uses the old cookies.txt file format rather than the sqlite database format that Firefox and Chrome currently use.

Here's what I would do:

Using Firefox or Chrome, go to the site and log in. (Make sure your browser is set to save cookies)
Quit your browser
Find your cookie file
Convert to cookies.txt format (see notes below on this)
wget --load-cookies cookies.txt http://forum.ubuntu-it.org/

Options to convert from sqlite format to cookies.txt include a python script or a simpler sqlite script (in the comments on that previous link), but the easiest for you might be to install this Firefox extension.

Doug Harris

Posted 2010-11-25T02:03:42.770

Reputation: 23 578