Google webmaster FAQs suggest that this will exclude all xml files from search:
User-agent: Googlebot
Disallow: /*.xml$
Is this legal for other bots as well?
User-agent: *
Disallow: /*.xml$
Asked
Active
Viewed 342 times
1 Answers
1
Yes, this is valid. Whether other bots actually understand and obey is completely up to them, though!
Shane Madden
- 112,982
- 12
- 174
- 248