0

I was just reading up on Yoast website here on how the correct robots.txt file should look like with the latest SEO practices. In their example Yoast uses the following:

User-Agent: *
Disallow: /suggest/?*

What exactly does line Disallow: /suggest/?* do?

Joe Bloggs
  • 101
  • Based of the answer below .. is there a particular reason why Yoast uses `/suggest/` in their robots file? Is this meant to be just a placeholder or does the `suggest` actually have a partiular meaning?? – Joe Bloggs Jun 10 '18 at 00:14

1 Answers1

1

What exactly does line Disallow: /suggest/?* do?

disallow and ?* will block search engines from accessing any URL that has a ? in it.

Altogether Disallow: /suggest/?* will block search engines from accessing urls such as yourwebsiteexample.com/suggest/?any.php=34

in other words, it will not block search engines from accessing other urls having ?

yourwebsiteexample.com/anyother/?any.php=34 <--- this will be crowed by search engines

more examples https://geoffkenyon.com/how-to-use-wildcards-robots-txt/

Aravinda
  • 1,081
  • 5
  • 12
  • 30
  • Based on that article, shouldn't it be `/*?` ? (Based on intuition I thought it was `/*?*`.) Otherwise, good explanation! – Esa Jokinen Jun 09 '18 at 15:17
  • Thanks @Aravinda and @EsaJonkien .. is there a particular reason why Yoast uses `/suggest/` in their robots file? Is this meant to be just a placeholder or does the `suggest` actually have a partiular meaning?? – Joe Bloggs Jun 10 '18 at 00:12
  • suggest should be example directory. no specific meaning. if you dont have one called suggest and if you dont have above requirement to block search engines from accessing any URL that has a ? , u don't have to use it – Aravinda Jun 10 '18 at 02:16