0

I have robots.txt that looks like this:

User-agent: *
Disallow: /system/
Disallow: /admin/
Disallow: /index.php

The obvious goal has been to prevent all the ugly URLs from being indexed, as they all begin with "/index.php".
But for some reason all URLs like /my-beautiful-sef-url-123 are listed under Crawl errors in Google Webmaster Tools with "URL restricted by robots.txt". (When I test such URL it yields Allowed for both Googlebot and Googlebot-Mobile.)

Can anyone help please?

1 Answers1

2

Are the URLs like /my-beautiful-sef-url-123 real pages or do they do a redirect to another page? If they do a redirect, maybe try testing the redirected to page in Google Webmaster??

user45348
  • 169
  • 3