How to find out the real download URL on download sites that use redirects

Question

Let's say I want to download something with wget but the website that has the files I need redirects to a site which automatically chooses a mirror for me (and there's no static file URL provided).

Downloading from such sites works with Firefox, but how do I get real url (not only the link to the redirect page) so I could download these files with wget?

Answer 1

When you've started the transfer in Firefox, cancel it, and right click the download and hit "Copy download link". If you're using an older version, right click it and click on "Properties", and the link will be in the window which opens.

In Chrome - run download as normal - then go to Menu - Downloads - and you should see the direct link which was used. Or press Ctrl + J to open the window.

Answer 2

You can use the LiveHTTPHeaders extension to determine the actual URL of the file being downloaded. (Keep an eye on the GETs in particular.)

Answer 3

While I like wget too, I use another similar tool: cURL. cURL specifically has a feature to follow redirects (-L / --location), and it's also free like wget. I suggest keeping both in your toolbelt; they have some complementary features.

Here's an interesting article I came across on how to use either wget or cURL to download from a site that uses cookies for authentication. There's a mention of the cURL --location feature to follow redirects. (In fact, based on no special flag mentioned for wget, it would seem to me that wget may follow redirects implicitly..?)

Answer 4

This is the answer you have been looking for!

There is only one fool-proof one-click solution: cliget

It gives you a curl command with headers, cookies and all, with a copy to clipboard button, right on the download dialog.

For Firefox: https://addons.mozilla.org/en-US/firefox/addon/cliget

I am not linking curlwget for Chrome here, because it's ugly. Please update this answer if Chrome gets an as elegant equivalent.

Answer 5

In those situations, First I starts the actual download and pause it and copy the URL.

Answer 6

I usually use the firefox DownThemAll addon when there are a lot of links to download and I need to select specific URLs.

It shows the full URL and allows you to add in paused mode so you can start the download when you like.

As an additional feature, you can grab the URL and skip the download altogether.
However, if you need to download the file, DownThemAll is quite good as an accelerator.

Answer 7

Can't you use wget directly using the --trust-server-name flag ? It will download and save using the filename it redirects to.

Answer 8

You can use Curl to download a redirected URL:

curl --location http://domainName/fileName 2> /dev/null > fileName

Answer 9

Your question is a bit unclear. There are 2 urls here : the one you use, and the one that came back after redirection. The first you should know, since it originated from your machine, while the second is the one that displayed in your browser. If the question pertains to where the file is on the redirected server, then there's no way to know.

If you're trying to debug the calls issued towards the server, you can use the firefox add-on of firebug, older versions of it exist here : https://addons.mozilla.org/en-US/firefox/addons/versions/1843.

Answer 10

You can use IDM (Internet Download Manager) when you click on the not direct download link it will redirect the link and showing up the direct link.

Answer 11

Have you tried to pass the redirect URL to wget? Sometimes they work for me.

Here a example from SourceForge:

$ wget http://sourceforge.net/projects/emule/files/eMule/0.49c/eMule0.49c-Installer.exe/download?use_mirror=ufpr,Centro%20de%20Computacao%20Cientifica%20e%20Software%20Livre
--14:23:20--  http://sourceforge.net/projects/emule/files/eMule/0.49c/eMule0.49c-Installer.exe/download?use_mirror=ufpr,Centro%20de%20Computacao%20Cientifica%20e%20Software%20Livre
Resolvendo sourceforge.net... 216.34.181.60
Connecting to sourceforge.net|216.34.181.60|:80... conectado!
HTTP requisição enviada, aguardando resposta... 302 Found
Localização: http://downloads.sourceforge.net/project/emule/eMule/0.49c/eMule0.49c-Installer.exe?use_mirror=ufpr [seguinte]
--14:23:21--  http://downloads.sourceforge.net/project/emule/eMule/0.49c/eMule0.49c-Installer.exe?use_mirror=ufpr
Resolvendo downloads.sourceforge.net... 216.34.181.59
Connecting to downloads.sourceforge.net|216.34.181.59|:80... conectado!
HTTP requisição enviada, aguardando resposta... 302 Found
Localização: http://ufpr.dl.sourceforge.net/project/emule/eMule/0.49c/eMule0.49c-Installer.exe [seguinte]
--14:23:21--  http://ufpr.dl.sourceforge.net/project/emule/eMule/0.49c/eMule0.49c-Installer.exe
Resolvendo ufpr.dl.sourceforge.net... 200.236.31.1, 200.17.202.1
Connecting to ufpr.dl.sourceforge.net|200.236.31.1|:80... conectado!
HTTP requisição enviada, aguardando resposta... 200 OK
Tamanho: 3342809 (3,2M) [application/x-msdos-program]
Saving to: `eMule0.49c-Installer.exe'

100%[====================================================================================>] 3.342.809   2,35M/s   in 1,4s

14:23:23 (2,35 MB/s) - `eMule0.49c-Installer.exe' saved [3342809/3342809]

The version I use:

$ wget --version
GNU Wget 1.10.2 

Answer 12

There was an addon for Firefox called Splitlink. It doesn't work for Firefox 3+, but it did work in Firefox 2. If you can find it anywhere (it's not on the official addons page anymore), it will give you more information about the real URL.

Answer 13

If you want a download manager to handle the download instead of firefox, you could use the Flashgot firefox addon.

Answer 14

The firefox copy url is definitely simplest. wget (as the question was tagged) is also a nice solution because you can see all the intermediate bounces, not just the final url:

$ wget 'http://sourceforge.net/projects/pdfcreator/files/PDFCreator/PDFCreator%200.9.8/PDFCreator-0_9_8_setup.exe/download'
--2009-09-10 09:59:53--  http://sourceforge.net/projects/pdfcreator/files/PDFCreator/PDFCreator%200.9.8/PDFCreator-0_9_8_setup.exe/download
Resolving sourceforge.net... 216.34.181.60
Connecting to sourceforge.net|216.34.181.60|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://downloads.sourceforge.net/project/pdfcreator/PDFCreator/PDFCreator%200.9.8/PDFCreator-0_9_8_setup.exe?use_mirror=voxel [following]
--2009-09-10 09:59:54--  http://downloads.sourceforge.net/project/pdfcreator/PDFCreator/PDFCreator%200.9.8/PDFCreator-0_9_8_setup.exe?use_mirror=voxel
Resolving downloads.sourceforge.net... 216.34.181.59
Connecting to downloads.sourceforge.net|216.34.181.59|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://voxel.dl.sourceforge.net/project/pdfcreator/PDFCreator/PDFCreator%200.9.8/PDFCreator-0_9_8_setup.exe [following]
--2009-09-10 09:59:54--  http://voxel.dl.sourceforge.net/project/pdfcreator/PDFCreator/PDFCreator%200.9.8/PDFCreator-0_9_8_setup.exe
Resolving voxel.dl.sourceforge.net... 74.63.52.167, 69.9.191.19, 69.9.191.18, ...
Connecting to voxel.dl.sourceforge.net|74.63.52.167|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 17695920 (17M) [application/octet-stream]
Saving to: `PDFCreator-0_9_8_setup.exe'

10% [=======> ] 1,818,064 609K/s

Answer 15

You could use the netcat tool (which is often included with linux distributions) on port 80 of the website. This shows the full raw HTTP response, including the headers which contain the redirect destination URL. This is one of the most direct ways to get the redirect location, since you are doing the same thing as downloaders do when they follow redirects. Here is an example for the URL http://www.example.com/example.

printf 'GET /example HTTP/1.1\r\nHost: www.example.com\r\n\r\n' | netcat www.example.com 80

...or...

Alternatively, many browsers show the actual download URL in the Downloads section. In these, you can start the download and cancel it right away, then copy the URL (from the download history).