Distributed web crawling

Distributed web crawling is a distributed computing technique whereby Internet search engines employ many computers to index the Internet via web crawling. Such systems may allow for users to voluntarily offer their own computing and bandwidth resources towards crawling web pages. By spreading the load of these tasks across many computers, costs that would otherwise be spent on maintaining large computing clusters are avoided.

Types

Cho[1] and Garcia-Molina studied two types of policies:

Dynamic assignment

With this type of policy, a central server assigns new URLs to different crawlers dynamically. This allows the central server to, for instance, dynamically balance the load of each crawler.

With dynamic assignment, typically the systems can also add or remove downloader processes. The central server may become the bottleneck, so most of the workload must be transferred to the distributed crawling processes for large crawls.

There are two configurations of crawling architectures with dynamic assignments that have been described by Shkapenyuk and Suel:[2]

  • A small crawler configuration, in which there is a central DNS resolver and central queues per Web site, and distributed downloaders.
  • A large crawler configuration, in which the DNS resolver and the queues are also distributed.

Static assignment

With this type of policy, there is a fixed rule stated from the beginning of the crawl that defines how to assign new URLs to the crawlers.

For static assignment, a hashing function can be used to transform URLs (or, even better, complete website names) into a number that corresponds to the index of the corresponding crawling process. As there are external links that will go from a Web site assigned to one crawling process to a website assigned to a different crawling process, some exchange of URLs must occur.

To reduce the overhead due to the exchange of URLs between crawling processes, the exchange should be done in batch, several URLs at a time, and the most cited URLs in the collection should be known by all crawling processes before the crawl (e.g.: using data from a previous crawl).[1]

Implementations

As of 2003, most modern commercial search engines use this technique. Google and Yahoo use thousands of individual computers to crawl the Web.

Newer projects are attempting to use a less structured, more ad hoc form of collaboration by enlisting volunteers to join the effort using, in many cases, their home or personal computers. LookSmart is the largest search engine to use this technique, which powers its Grub distributed web-crawling project.

This solution uses computers that are connected to the Internet to crawl Internet addresses in the background. Upon downloading crawled web pages, they are compressed and sent back together with a status flag (e.g. changed, new, down, redirected) to the powerful central servers. The servers, which manage a large database, send out new URLs to clients for testing.

Drawbacks

According to the FAQ about Nutch, an open-source search engine website, the savings in bandwidth by distributed web crawling are not significant, since "A successful search engine requires more bandwidth to upload query result pages than its crawler needs to download pages...".

gollark: guys, what if lemmmy *is* yemmel?
gollark: Once PotatOS for x86 is installed on Terra's server, it will use my arbitrary code execution exploit in *string theory* to teleport the server to my house, where it will then run Minecraft for me.
gollark: "I'm bad, but other people are worse! Ignore me and complain at them!"
gollark: Soon the dead-man's switch in the bunker will trigger and disassemble Terra's bunker, then most of itself, then use an arbitrary code execution exploit in Minecraft to install PotatOS for x86 on all Terra's devices, then it will crash TC2020 forever.
gollark: I built a bunker there and everything. Well, half a bunker. I haven't finished the self-replication systems.

See also

Sources
  1. Cho, Junghoo; Garcia-Molina, Hector (2002). "Parallel crawlers". Proceedings of the 11th international conference on World Wide Web. ACM. pp. 124–135. doi:10.1145/511446.511464. ISBN 1-58113-449-5. Retrieved 2015-10-13.
  2. Shkapenyuk, Vladislav; Suel, Torsten (2002). "Design and implementation of a high-performance distributed web crawler". Data Engineering, 2002. Proceedings. 18th International Conference on. IEEE. pp. 357–368. Retrieved 2015-10-13.
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.