How to download a URL from internet by netcat?

Question

I can use netcat to transfer files between two computers.

# Server
nc -l -p 8080 < file

# client
nc 192.168.1.101 8080 > file

---

Can I use netcat to download a URL from Internet like wget?

Answer 1

Assuming you mean a regular HTTP URL, it's possible but it requires you to manually type in the HTTP Request. If the URL requires an HTTP POST with a url-encoded form or looks for specific headers, this can be quite cumbersome.

You will need to supply a valid HTTP Request to the stdin of netcat. The general idea is:

Execute the command nc example.com 80

netcat will start reading from stdin, and whatever you type in or paste into the terminal will be sent over the network to the remote server on the port you selected (80 is the default HTTP port).

Now you need to type in or paste an HTTP request, like:

GET / HTTP/1.1
Host: example.com
User-Agent: netcat

You'll also need some newlines at the end, so press enter repeatedly until you get the HTTP response back. Then you'll have to manually parse the response.

netcat works at the TCP layer, so any protocol based on HTTP can technically be used with netcat, but for protocols that don't suit to human input (such as ssh and https), it can be difficult to use if you aren't scripting it from a program. And if you ARE writing a program, why aren't you just using normal TCP sockets?

The limitations of netcat for this purpose are as follows:

• netcat doesn't understand HTTP, or FTP, or HTTPS, or SSH, or any other application-layer protocol. Whatever script, program, or manual keying you use, that input will have to have sufficient knowledge of the application-layer wire protocol in order to properly interface with the server.
• Even simple protocols like HTTP can have complications that require detailed knowledge of the protocol -- chunked encoding, url-encoded forms, different behavior depending on supplied cookies, user agent, etc, and HTTPS requiring a successful SSL handshake. I don't know of anyone who can manually key in an SSL handshake using only their brain :)
• Since netcat is dumb, it provides no advantages over just using a programmatic TCP socket if you are writing a program. If you're just typing on the terminal, it's easier to use a program that supports proper application-layer protocols, such as wget, curl, lftp, ssh, and so on.

Answer 2

# create a request file
$ cat req
GET / HTTP/1.1
Host: superuser.com
User-Agent: Mozilla/5.0

# send request to get response
$ nc superuser.com 80 <req >res