Strange Apache log entries coming from a Google IP address

Question

I have this kind of error messages in my apache error log file:

[Wed Sep 27 18:45:34.289807 2017] [:error] [pid 17543] [client 66.249.79.21:47149] PHP Notice:  Undefined variable: catNameUrl in /var/www/dev/page/page_display.inc.php on line 73
[Wed Sep 27 18:59:11.964934 2017] [:error] [pid 18906] [client 66.249.79.19:58443] PHP Warning:  getimagesize(../include/images/sizes/3GHI9W6AyEI5/--1000.jpg): failed to open stream: No such file or directory in /var/www/dev/page/index_framed.php on line 66

And 1000 times per hour:

[Wed Sep 27 18:59:11.973506 2017] [:error] [pid 18906] [client 66.249.79.19:58443] PHP Warning:  Division by zero in /var/www/dev/include/php/global_images.inc.php on line 335
[Wed Sep 27 18:59:11.973729 2017] [:error] [pid 18906] [client 66.249.79.19:58443] PHP Warning:  Division by zero in /var/www/dev/include/php/global_images.inc.php on line 335
[Wed Sep 27 18:59:11.973878 2017] [:error] [pid 18906] [client 66.249.79.19:58443] PHP Warning:  Division by zero in /var/www/dev/include/php/global_images.inc.php on line 335
[Wed Sep 27 18:59:11.974017 2017] [:error] [pid 18906] [client 66.249.79.19:58443] PHP Warning:  Division by zero in /var/www/dev/include/php/global_images.inc.php on line 335

etc...

The fact is, there is no referer about the page requested. This is a normal error log:

[Wed Sep 27 09:09:39.877580 2017] [:error] [pid 29730] [client XX.XX.XX.XX:18888] PHP Notice:  Undefined index: cat in /var/www/dev/page/page_display.inc.php on line 441, referer: https://dev.mywebsite.com/page/redaction/p/10

https://www.abuseipdb.com/check/66.249.79.17 -> Google Search Engine Spider

I think, it's opening wrong urls and then can't open a file which is used to create dynamically an image.

Why there is no referer in the log when this ip reach my dev website?

Answer 1

Why there is no referer in the log when this IP reaches my dev website?

Bots, including Googlebot, do not typically send referer headers.

Another consideration is that there are no referer headers to send if the page is accessed directly (referer headers are typically blank in these cases unless programmed not to be).

More generally, you might be interested in this Webmasters question, which has a good list of other reasons why referer headers aren't sent (though the two listed above are the ones most likely to apply).

Answer 2

This is most likely a Google crawler attempting to index your site‘s content. So I wouldn’t worry about it. That said, you state this:

And 1000 times per hour…

And provide these lines as an example:

[Wed Sep 27 18:59:11.973506 2017] [:error] [pid 18906] [client 66.249.79.19:58443] PHP Warning:  Division by zero in /var/www/dev/include/php/global_images.inc.php on line 335
[Wed Sep 27 18:59:11.973729 2017] [:error] [pid 18906] [client 66.249.79.19:58443] PHP Warning:  Division by zero in /var/www/dev/include/php/global_images.inc.php on line 335
[Wed Sep 27 18:59:11.973878 2017] [:error] [pid 18906] [client 66.249.79.19:58443] PHP Warning:  Division by zero in /var/www/dev/include/php/global_images.inc.php on line 335
[Wed Sep 27 18:59:11.974017 2017] [:error] [pid 18906] [client 66.249.79.19:58443] PHP Warning:  Division by zero in /var/www/dev/include/php/global_images.inc.php on line 335

The key part is the “PHP Warning: Division by zero…” since that is an error in your site’s code and scripting. Your two choices are to disable PHP warnings on your production server which will get rid of those error messages or—my preferred solution—would be to open up global_images.inc.php, go to line 335 and figure out what is causing a division by zero error and patch that.

You will never be able to fully control the way a bot crawls your site no matter what you say. But if you want to limit noise in your error logs, that is fully in your control. Clean up the code or throttle back on “warning” level error on your site and don’t lose any more sleep over this than you need to.