7

I've started a relatively new website, and I submitted it to google and everything. I use google's webmaster tools as well. I'm wondering how to figure out the frequency of google's spider accessing my website. I always hear people talking in forums about "I'm getting 250 pages crawled an hour" and so on... Is there software to use? A javascript method similar to google analytics? Or is there something to analyze my server's logs? Thanks

8 Answers8

4

Just check the webserver logs if the visitor's User-Agent request header matches/contains Googlebot. There are lot of webserver log analyzer tools, either free or payware. Most of them are also able to categorize bots. From them all I've had the best experience with Google Analytics.

  • Ah, thanks. Can you tell me what the log file will probably be named? I'm using a 1and1 server and I'm fairly new to the web hosting game –  Jan 23 '10 at 23:17
  • You're using 3rd party hosting? Well, contact them for more details or poke a bit round in the hosting's control panel. –  Jan 23 '10 at 23:18
  • By the way, with Google Analytics you don't need to know about webserver logs at all. It runs completely independently. –  Jan 23 '10 at 23:33
2

Signup for Google Webmaster Central, aka Webmasters Tools, verify your site, and wait for the stats to appear.

Pentium10
  • 434
  • 9
  • 23
1

It's fairly unlikely Google Bot will run any Javascript on your site that will track it arriving, so the best hope is to look at the site logs themselves - either in webalizer or the hundreds of other log file parsers, or just open them up in a texteditor and run a regex to extract the Google Bot user agent.

You should see then how many hits it makes in a space of time. If it's too much you might want to look at using an XML sitemap

Chris S
  • 113
  • 4
  • Yeah, I didn't think it would run any javascript. Thanks for the suggestions –  Jan 23 '10 at 23:19
1

AWstats is a good, free utility to use for log file analysis. It'll give you a lot of interesting statistics in addition to the Google crawl info.

Mahmoud Al-Qudsi
  • 509
  • 1
  • 6
  • 21
0

I would recommend Google analytics, it covers all visits to your site and you can work out whether a visitor was a bot or not. It is also excellent for SEO on your site helping to increase visitors.

Burt
  • 160
  • 1
  • 1
  • 5
  • 1
    Google Analytics doesn't include bots because it uses javascript, which bots generally don't execute. – hmallett Jan 24 '10 at 00:34
0

AWstats is a good, free utility to use for log file analysis. It'll give you a lot of interesting statistics in addition to the Google crawl info.

Awstats is a great tool indeed

Camy
  • 11
  • 1
0

Check this article which tells how to send mail when Googlebot crawls a webpage of your website

0

You can use googles webmaster tools, It will tell you how often the bot has crawled your site...

Jacob
  • 9,114
  • 4
  • 44
  • 56