4
1
Hi trying to determine all the valid urls under a given domain without having the mirror the site locally.
People generally want to download all the pages but I just want to get a list of the direct urls under a given domain (e.g. www.example.com), which would be something like www.example.com/page1, www.example.com/page2, etc.
Is there a way to use wget to do this? or is there a better tool for this?
2In order to determine the links on each page, you will need to see the page (I.e. Download it) – Brian Adkins – 2013-09-24T18:59:41.680
@BrianAdkins: I am Ok with downloading but I woant to keep only the urls not the pages' contents – fccoelho – 2013-09-24T19:05:16.377
There's a
--spider
option that downloads the page, but doesn't save it. – LawrenceC – 2013-09-25T22:43:40.190