Googlebot incrementing page id

Question

So here is an example of a hit I'm getting from the googlebot:

66.249.73.171 - - [19/Feb/2013:16:12:39 -0500] "GET /eghm-blah.php?pid=2855 HTTP/1.1" 200 1684 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

My posts are setup to increment the pid for new posts...however the highest pid I have is 13. The googlebot has been slowly incrementing the page numbers and looking for new posts. When it makes the access attempt it will not 404 as eghm-blah.php will display a message that it can't find your post.

Is there anyway to stop googlebot from hitting my server multiple times per day looking for pid's which don't exist?

Make `eghm-blah.php` correctly 404 when the post doesn't exist. — mgorven, Feb 19 '13 at 21:22

score 3 · Accepted Answer · answered Feb 19 '13 at 21:25

3

eghm-blah.php should be setting the return code to 404. You can still display whatever you want on the page but the return code needs to be 404.

answered Feb 19 '13 at 21:25

longneck

22,793
4
50
84

Ah ok. I thought there may have been a way to stop it without throwing a 404. – BOMEz Feb 19 '13 at 22:09
You also likely have some reference to pid+1 on your 404 page. Google is likely following that – R. S. Feb 20 '13 at 00:30

Googlebot incrementing page id

1 Answers1