Yes, it is possible. Sometimes.
When you browse to a webpage (Say to http://demo.domain.tld/testdir/index.html
) it will open the file you specified (in this case `index.html).
If you do not specify a file and there is a default present (e.g. the web-server is configured to show index.html, or index.php, ...) and you typed http://demo.domain.tld/testdir/
then it will automagically present you with the right file.
If that file is not present then it can do other things, such as listing the directory contents. This is quite useful when building a site, however it is also considered unwise from a security standpoint.
TL;DR: Yes, sometimes it is possible.
However the more practical approach is to simply SSH, [s]FTP or RDP to the web-server and issue a local directory listing command.
+1 for the ``--spider
option. I did not know that one thogh I have used
wgetand
curl` before. – Hennes – 2013-09-08T14:25:14.387