Downloading XCode using wget or curl

Is there a way for downloading XCode using curl or wget?

EDIT: Why is the following curl snippet not working?

First, I login to Apple's site:

curl -d "theAccountName=USERNAME&theAccountPW=PASS" -c xcode-cookie 
-A "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1" 
https://daw.apple.com//cgi-bin/WebObjects/DSAuthWeb.woa/104/wo/W1df7WQblBsLekQzvS1tJ0/0.5.3.1.1.2.1.1.3.1.1"

Then, I try to download the file:

curl -b xcode-cookie -c xcode-cookie -O -v -A 
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"
http://adcdownload.apple.com/ios/ios_sdk_4.0.2__final/xcode_3.2.3_and_ios_sdk_4.0.2.dmg

However I get a 403 Forbidden error:

* About to connect() to adcdownload.apple.com port 80 (#0)
*   Trying 60.254.148.16...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0connected
* Connected to adcdownload.apple.com (60.254.148.16) port 80 (#0)
> GET /ios/ios_sdk_4.0.2__final/xcode_3.2.3_and_ios_sdk_4.0.2.dmg HTTP/1.1
> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1
> Host: adcdownload.apple.com
> Accept: */*
> Cookie: myacinfo=foo; DefaultAppleID=bar; ds01=baz
> 
< HTTP/1.1 403 Forbidden
< Server: Apache-Coyote/1.1
< X-Powered-By: Servlet 2.4; JBoss-4.0.5.GA (build: CVSTag=Branch_4_0 date=200610162339)/Tomcat-5.5
< no-cache: Cache-Control
< no-store: Cache-Control
< Content-Type: text/html;charset=ISO-8859-1
< Date: Mon, 30 Aug 2010 05:30:42 GMT
< Transfer-Encoding:  chunked
< Connection: keep-alive
< Connection: Transfer-Encoding
< Cache-Control: no-store
< 
{ [data not shown]
100  1932    0  1932    0     0   7531      0 --:--:-- --:--:-- --:--:--  8548* Connection #0 to host adcdownload.apple.com left intact

* Closing connection #0

Behrang Saeedzadeh

Posted 2010-08-25T12:04:55.937

Reputation: 1 834

Answers

Josh thinks too simple-mindedly. The URL to download Xcode is indeed http://developer.apple.com/iphone/download.action?path=%2Fios%2Fios_sdk_4.0.2__final%2Fxcode_3.2.3_and_ios_sdk_4.0.2.dmg

However, you must be authenticated to download the file.

Now both Wget and cURL support cookies, but I've never used cURL for downloads so here's a couple of lines using Wget to first authenticate into the developer site (storing the authentication cookies in a cookies.txt file), then another line to actually grab the download (recalling that same cookies.txt file):

wget --post-data="theAccountName=YOURUSERNAME&theAccountPW=YOURPASSWORD" --cookies=on --load-cookies=cookies.txt --keep-session-cookies --save-cookies=cookies.txt https://daw.apple.com/cgi-bin/WebObjects/DSAuthWeb.woa/67/wo/99OWdOWyhWA2pMdrPemdSw/0.5.3.1.1.2.1.1.3.1.1
wget --cookies=on --load-cookies=cookies.txt --keep-session-cookies --save-cookies=cookies.txt http://developer.apple.com/iphone/download.action?path=%2Fios%2Fios_sdk_4.0.2__final%2Fxcode_3.2.3_and_ios_sdk_4.0.2.dmg

Be sure to change YOURUSERNAME and YOURPASSWORD to the appropriate values, and change the path to the cookies.txt file if so desired. Also, note the URL of the login form might change, so be sure to check the action attribute of the login form from your browser if this fails to work.

I tested this on my server, and the download took about 40 minutes. It worked a treat!

purefusion

Posted 2010-08-25T12:04:55.937

Reputation: 687

Sure, I didn't explicitly mention that, it was assumed if you need authentication you can use the options available for the tool. +1, nice that you pasted that code in for him. – Josh K – 2010-08-25T19:02:21.677

On here, people tend to expect to be shown some code. :) – purefusion – 2010-08-25T19:05:02.037

Thanks, but the problem is that OS X does not come bundled with wget and I need to use curl. Please see my updated post. I have accepted your solution but a curl solution will be highly appreciated. – Behrang Saeedzadeh – 2010-08-30T05:48:19.450

Single command, using stock Mac OS X 10.6.6 command line tools (bash, curl and sqlite3).

curl -b <(sqlite3 -separator $'\t' ~/Library/Application\ Support/Google/Chrome/Default/Cookies "select host_key, 'TRUE','/', 'FALSE', expires_utc, name, value from cookies where host_key like '%apple.com%'") http://adcdownload.apple.com/ios/ios_sdk_4.2__final/xcode_3.2.5_and_ios_sdk_4.2_final.dmg

Caveats:

This uses a query of Google Chrome's sqlite cookie database.
You need to login via Apple's Xcode developer page with the browser first so that the cookies get set.

Notes:

<() is bash's Process Substitution operator and does away with the instantly-stale temporary cookies file.
If, like me, you are resuming a stalled browser download, and you have the file, use these arguments:
```
-C - -o unconfirmed\ 21043.crdownload
```

Now that you have Xcode, you can install Homebrew, and then wget. ;)

vdm

Posted 2010-08-25T12:04:55.937

Reputation: 191

1Hehe. Of course, then he'd already have XCode and would no longer need wget to download it (save for perhaps future updates). I wasn't aware wget didn't come with OSX though. How'd I miss that? I guess I've never tried to use it on anything but my remote linux server. – purefusion – 2011-01-21T06:21:18.140

Of course, just copy the download link that you got from Apple.

Josh K

Posted 2010-08-25T12:04:55.937

Reputation: 11 754

2It needs authentication. – Behrang Saeedzadeh – 2010-08-25T15:41:31.350

@bytecode: Hence the availability of --post-data and --cookie options for wget. You can even do those things in curl. – Josh K – 2010-08-25T19:01:19.320