I have this robots.txt file:
User-agent: *
Sitemap: Path_to_sitemap.xml
My Q is, should I have something else in there as well? Like allow All or something?
Thanks
I have this robots.txt file:
User-agent: *
Sitemap: Path_to_sitemap.xml
My Q is, should I have something else in there as well? Like allow All or something?
Thanks
It is a valid robots.txt. You may read this post from google: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449
Depends on what you want ;-) You don´t have to allow sth. special, just block the things you wouldn´t have crawled by Google and Co.