making sense of wget -r output

Question

This is the output of tree command in one directory:

.
|-- asdf.txt
|-- asd.txt
|-- fabc
|   |-- fbca
|   `-- file1.txt
|-- fldr1
|-- fldr2
|   `-- index.html
|-- fldr3
|   |-- cap.txt
|   `-- f01
`-- out.txt

6 directories, 6 files

I start a local http server in this directory. Next I run the following command:

wget -r -nv --spider --no-parent http://localhost:3000 -o -

...and get the following output:

2017-01-02 20:07:24 URL:http://localhost:3000/ [1580] -> "localhost:3000/index.html" [1]
http://localhost:3000/robots.txt:
2017-01-02 20:07:24 ERROR 404: Not Found.
2017-01-02 20:07:24 URL:http://localhost:3000/fabc/ [897] -> "localhost:3000/fabc/index.html" [1]
2017-01-02 20:07:24 URL:http://localhost:3000/fldr1/ [536] -> "localhost:3000/fldr1/index.html" [1]
2017-01-02 20:07:24 URL:http://localhost:3000/fldr2/ [0/0] -> "localhost:3000/fldr2/index.html" [1]
2017-01-02 20:07:24 URL:http://localhost:3000/fldr3/ [896] -> "localhost:3000/fldr3/index.html" [1]
2017-01-02 20:07:24 URL: http://localhost:3000/asd.txt 200 OK
unlink: No such file or directory
2017-01-02 20:07:24 URL: http://localhost:3000/asdf.txt 200 OK
unlink: No such file or directory
2017-01-02 20:07:24 URL: http://localhost:3000/out.txt 200 OK
unlink: No such file or directory
2017-01-02 20:07:24 URL:http://localhost:3000/fabc/fbca/ [548] -> "localhost:3000/fabc/fbca/index.html" [1]
2017-01-02 20:07:24 URL: http://localhost:3000/fabc/file1.txt 200 OK
unlink: No such file or directory
2017-01-02 20:07:24 URL:http://localhost:3000/fldr3/f01/ [548] -> "localhost:3000/fldr3/f01/index.html" [1]
2017-01-02 20:07:24 URL: http://localhost:3000/fldr3/cap.txt 200 OK
unlink: No such file or directory
Found no broken links.

FINISHED --2017-01-02 20:07:24--
Total wall clock time: 0.3s
Downloaded: 7 files, 4.9K in 0s (43.4 MB/s)

1. Is wget written to always seek index.html? Can we disable this?
2. What are those numbers such as 1580, 536, 0/0, etc?
3. Why does it say unlink: No such file or directory?

Answer 1

1. You can try to skip over files with --reject option (accepts wildcards as well):

   wget --reject index.html

However you don't want to do this. When using wget with -r, it somehow needs to get a list of files inside the directory. Thus wget asks for the index.html file and parses the content in hope to get paths to other files in this directory. When there is no index.html file in the folder, the webserver will usually generate it for wget - this file will contain the directory listing. The creation of this list file has to be enabled on the webserver - otherwise wget will receive a HTTP 404 reply and fail with the recursive download.

2. This is the file size in bytes.
3. This means that a file couldn't be removed (probably because it wasn't created in first place). Do you have write permission on the directory into which you download with wget?

Edit: After testing wget downloads with --spider and --recursive I've reproduced your unlink error. It seems that wget uses the content type of response to determine if the file can contain links to other resources. If content type test fails and file is not downloaded, wget will still try to remove the temporary file, as if it was downloaded (This is apparent when rerunning wget with --debug. It will clearly state Removing file due to --spider in recursive_retrieve():). I guess you've found a bug in wget.