Website crawler/spider to get site map

Question

I need to retrieve a whole website map, in a format like :

• http://example.org/
• http://example.org/product/
• http://example.org/service/
• http://example.org/about/
• http://example.org/product/viewproduct/

I need it to be linked-based (no file or dir brute-force), like :

parse homepage -> retrieve all links -> explore them -> retrieve links, ...

And I also need the ability to detect if a page is a "template" to not retrieve all of the "child-pages". For example if the following links are found :

• http://example.org/product/viewproduct?id=1
• http://example.org/product/viewproduct?id=2
• http://example.org/product/viewproduct?id=3

I need to get only once the http://example.org/product/viewproduct

I've looked into HTTtracks, wget (with spider-option), but nothing conclusive so far.

The soft/tool should be downloadable, and I prefer if it runs on Linux. It can be written in any language.

Thanks

Answer 1

After a lot of research no tool has satisfied me, therefore I'm coding my own using http://scrapy.org/doc/

Answer 2

Here is an example of one made in python:

(Taken from http://theanti9.wordpress.com/2009/02/14/python-web-crawler-in-less-than-50-lines/ )

Also on that website there is a link to a github project http://github.com/theanti9/PyCrawler that is a more robust version the person made.

import sys
import re
import urllib2
import urlparse
tocrawl = set(["http://www.facebook.com/"])
crawled = set([])
keywordregex = re.compile('<meta\sname=["\']keywords["\']\scontent=["\'](.*?)["\']\s/>')
linkregex = re.compile('<a\s*href=[\'|"](.*?)[\'"].*?>')

while 1:
    try:
        crawling = tocrawl.pop()
        print crawling
    except KeyError:
        raise StopIteration
    url = urlparse.urlparse(crawling)
    try:
        response = urllib2.urlopen(crawling)
    except:
        continue
    msg = response.read()
    startPos = msg.find('<title>')
    if startPos != -1:
        endPos = msg.find('</title>', startPos+7)
        if endPos != -1:
            title = msg[startPos+7:endPos]
            print title
    keywordlist = keywordregex.findall(msg)
    if len(keywordlist) > 0:
        keywordlist = keywordlist[0]
        keywordlist = keywordlist.split(", ")
        print keywordlist
    links = linkregex.findall(msg)
    crawled.add(crawling)
    for link in (links.pop(0) for _ in xrange(len(links))):
        if link.startswith('/'):
            link = 'http://' + url[1] + link
        elif link.startswith('#'):
            link = 'http://' + url[1] + url[2] + link
        elif not link.startswith('http'):
            link = 'http://' + url[1] + '/' + link
        if link not in crawled:
            tocrawl.add(link)

Answer 3

I personnaly use Kapow Katalyst, but I guess it's out of your budget. If not, it's probably the most intuitive software to create spiders, and much more if you need.

Answer 4

Technically speaking there is no foolproof way of extracting the directory structure of a website.

This is because HTTP is not a network file system. The only thing you can do with HTTP is follow the links from the starting page. Furthermore, there's nothing that requires the starting page to have links only to its immediate subdirectory. A top level index.html page may, for example, have a direct link to "foo/baz/blah.html", deep in some subdirectory.

Edit:

• To generate basic site maps, Some online tools are there commonly known as Sitemap Generator. One such tool is web-site-map.com, it gives sitemap in XML.

• If you are comfortable with programming then you can write your own web-spider, with specific set of rules of a particular site.

Answer 5

(Win)HTTrack does a very decent job.

It allows you to download a World Wide Web site from the Internet to a local directory, building recursively all directories, getting HTML, images, and other files from the server to your computer.