1

When I try to fetch my site as a googlebot in webmaster tools it return Unreachable robots.txt, after investigate I understood google bot can see my server:

tcpdump | grep google

It returns that google can access my server with IP aa.bb.cc.xx or aa.bb.cc.yy. but there is not anything in access_log or error_log or other apache logs.

cat access_log | grep google or cat error_log | grep aa.bb.cc.xx

Other bot (bing,...) can access apache but google can't.

There is no problem in my robots.txt or its permissions because as you know robots.txt is not necessary so I delete it but again webmaster tools returned "Unreachable robots.txt", not 404 not found!

Information about server:

  • Server OS : CentOS 6
  • Web Server : Apache 2.x
  • Firewall : IPTables is stoped
  • SELinux is Disabled
  • There is not any think else for security on my server.

How can I investigate the problem and is there any other command that can help me to find the problem ?

mveroone
  • 447
  • 7
  • 22
Ahmad Ahmadi
  • 11
  • 1

1 Answers1

2

You deleted the robots.txt file, of course it's "unreachable" - the 404 error that Google's getting in response to the request for it is because the file doesn't exist.

If the tool requires a robots.txt to exist, then you probably need one.

Shane Madden
  • 112,982
  • 12
  • 174
  • 248